Getting Data In

Is there a way to test index-time operations without indexing logs?

EatMoreChicken
Explorer

Is there a way to test index-time operations without indexing logs? For example, is there a way I can provide a sample log file and see what the timestamp, host, sourcetype, source, and output after other operations like null-queuing would be?

For example, I currently use the "Add Data" section to test timestamping and line-breaking, but this doesn't show other metadata or what will be ingested after null-queuing.

I also setup a quick bash command to make copies of the base log samples and have inputs continuously monitor the new files as I'm testing new sourcetypes. I feel like this is a bit inefficient.

Thanks in advance for any input!

0 Karma

PickleRick
Ultra Champion

Well, the "proper" process would be to have a test environment anyway. If I remember correctly, you can get a free dev/test license for the testing purposses (it has limited functionality however). You can also create a small license pool from your main license and allocate it to a test environment.

That would be the "proper" solution.

You can also just have a test index and test configurations for test sources so you'd just ingest data, verify if everything's ok, then just delete index and create it anew.

Unfortunately, there's no way to do the ingestion process without ingesting data 😉

0 Karma

EatMoreChicken
Explorer

Yep, the Dev license is what I use at the moment in a dev environment. I appreciate the input, it looks like the long way is the only way at the moment sadly. 😢

0 Karma

gcusello
Legend

Hi @EatMoreChicken,

I usually test my ingestion taking an offline sample of the logs to ingest and ingesting it using the web Add Data feature.

In this way you can test the sourcetype (timestamp recognition, event breaking, etc...) before indexing.

Ciao.

Giuseppe

 

EatMoreChicken
Explorer

Whoops, meant to say "Add Data" in my original post. But yes, this is also the process I use at the moment. The only issue with this method is that I'm not able to see how the host, source, and null-queuing is affected without actually indexing the data.

0 Karma

gcusello
Legend

Hi @EatMoreChicken,

host and source are usually defined by the inputs.conf on the target systems so they usually aren't the main problem to test.

About null-queuing, you can test it putting the filtering conditions also on the system that you're using for the test.

If you don't want to dirty your production Search Heads, you could perform the test on a test system, containing props.conf and transforms.conf used in the filtering.

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...