topic Re: Is there a way to test index-time operations without indexing logs? in Getting Data In

Is there a way to test index-time operations without indexing logs?

EatMoreChicken — Wed, 06 Apr 2022 17:21:18 GMT

Is there a way to test index-time operations without indexing logs? For example, is there a way I can provide a sample log file and see what the timestamp, host, sourcetype, source, and output after other operations like null-queuing would be?

For example, I currently use the "Add Data" section to test timestamping and line-breaking, but this doesn't show other metadata or what will be ingested after null-queuing.

I also setup a quick bash command to make copies of the base log samples and have inputs continuously monitor the new files as I'm testing new sourcetypes. I feel like this is a bit inefficient.

Thanks in advance for any input!

Re: Testing index-time operations without indexing logs

gcusello — Wed, 06 Apr 2022 15:22:04 GMT

Hi @EatMoreChicken,

I usually test my ingestion taking an offline sample of the logs to ingest and ingesting it using the web Add Data feature.

In this way you can test the sourcetype (timestamp recognition, event breaking, etc...) before indexing.

Ciao.

Giuseppe

Re: Testing index-time operations without indexing logs

EatMoreChicken — Wed, 06 Apr 2022 17:21:03 GMT

Whoops, meant to say "Add Data" in my original post. But yes, this is also the process I use at the moment. The only issue with this method is that I'm not able to see how the host, source, and null-queuing is affected without actually indexing the data.

Re: Testing index-time operations without indexing logs

gcusello — Thu, 07 Apr 2022 06:35:39 GMT

Hi @EatMoreChicken,

host and source are usually defined by the inputs.conf on the target systems so they usually aren't the main problem to test.

About null-queuing, you can test it putting the filtering conditions also on the system that you're using for the test.

If you don't want to dirty your production Search Heads, you could perform the test on a test system, containing props.conf and transforms.conf used in the filtering.

Ciao.

Giuseppe

Re: Is there a way to test index-time operations without indexing logs?

PickleRick — Thu, 07 Apr 2022 07:35:52 GMT

Well, the "proper" process would be to have a test environment anyway. If I remember correctly, you can get a free dev/test license for the testing purposses (it has limited functionality however). You can also create a small license pool from your main license and allocate it to a test environment.

That would be the "proper" solution.

You can also just have a test index and test configurations for test sources so you'd just ingest data, verify if everything's ok, then just delete index and create it anew.

Unfortunately, there's no way to do the ingestion process without ingesting data 😉

Re: Is there a way to test index-time operations without indexing logs?

EatMoreChicken — Thu, 07 Apr 2022 12:14:09 GMT

Yep, the Dev license is what I use at the moment in a dev environment. I appreciate the input, it looks like the long way is the only way at the moment sadly. 😢