Getting Data In

Splunk on-demand indexing of a file

batcave
Explorer

I currently have a FIX log file which generates HUGE amounts of data every day. With my current license its impossible to index all that data.

Is it possible to get parts of the log file on-demand and then index and query them as and when needed? Is it possible to achieve this via a script invoked from a Splunk query?

Tags (2)
0 Karma

batcave
Explorer

or basically any way to directly query data without indexing it in Splunk?

0 Karma

rtadams89
Contributor

Also, see routing and filtering data, specifically to a null queue: http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Routeandfilterdatad

0 Karma

rtadams89
Contributor
0 Karma

Ayn
Legend

I would say no to your question - there might in certain situations be some kind of hack that makes you do this but in essence this is not how Splunk was meant to work.

0 Karma

batcave
Explorer

"Currently, it is not supported by Splunk"-- Not sure whether this command works anymore. any ideas guys?

0 Karma

rtadams89
Contributor

Possibly, but only if the data to index is located on the Splunk search head (or is reachable via a script on the search head). See http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Runshellscript . This will run a script on the search head.

0 Karma

batcave
Explorer

ok sounds good in terms of permissions but as you mentioned this can be scheduled to run via cron or task scheduler. What I need is to run it on demand anytime and gat her data only from a particular time period(which I think can be done via arguements to the script). Bu thow to invoke the script on demand rather than on a schedule?

0 Karma

rtadams89
Contributor

Scripted inputs run under the context of the Splunk user. If that doesn't work for your specific needs, you could create a script that is run via Windows task scheduler (which will let you select the user to run the script as). The script just needs to do what ever parsing of the original log file you want, and then output it to a new file that you can have Splunk monitor/index.

0 Karma

batcave
Explorer

scripted input don't think would fit the case as the script should be invokable via non admin users as well, which I am not sure can be done

0 Karma
Get Updates on the Splunk Community!

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...