topic Re: Splunk on-demand indexing of a file in Getting Data In

Splunk on-demand indexing of a file

batcave — Tue, 29 Jan 2013 04:07:22 GMT

I currently have a FIX log file which generates HUGE amounts of data every day. With my current license its impossible to index all that data.

Is it possible to get parts of the log file on-demand and then index and query them as and when needed? Is it possible to achieve this via a script invoked from a Splunk query?

Re: Splunk on-demand indexing of a file

rtadams89 — Tue, 29 Jan 2013 04:48:04 GMT

Yes, you can use scripted input (http://docs.splunk.com/Documentation/Splunk/5.0.1/AdvancedDev/ScriptSetup).

Re: Splunk on-demand indexing of a file

batcave — Tue, 29 Jan 2013 05:52:35 GMT

scripted input don't think would fit the case as the script should be invokable via non admin users as well, which I am not sure can be done

Re: Splunk on-demand indexing of a file

rtadams89 — Tue, 29 Jan 2013 05:58:35 GMT

Scripted inputs run under the context of the Splunk user. If that doesn't work for your specific needs, you could create a script that is run via Windows task scheduler (which will let you select the user to run the script as). The script just needs to do what ever parsing of the original log file you want, and then output it to a new file that you can have Splunk monitor/index.

Re: Splunk on-demand indexing of a file

rtadams89 — Tue, 29 Jan 2013 05:59:35 GMT

Also, see routing and filtering data, specifically to a null queue: http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Routeandfilterdatad

Re: Splunk on-demand indexing of a file

batcave — Tue, 29 Jan 2013 06:06:56 GMT

or basically any way to directly query data without indexing it in Splunk?

Re: Splunk on-demand indexing of a file

batcave — Tue, 29 Jan 2013 06:15:35 GMT

ok sounds good in terms of permissions but as you mentioned this can be scheduled to run via cron or task scheduler. What I need is to run it on demand anytime and gat her data only from a particular time period(which I think can be done via arguements to the script). Bu thow to invoke the script on demand rather than on a schedule?

Re: Splunk on-demand indexing of a file

rtadams89 — Tue, 29 Jan 2013 06:25:03 GMT

Possibly, but only if the data to index is located on the Splunk search head (or is reachable via a script on the search head). See http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Runshellscript . This will run a script on the search head.

Re: Splunk on-demand indexing of a file

batcave — Wed, 06 Feb 2013 06:15:23 GMT

"Currently, it is not supported by Splunk"-- Not sure whether this command works anymore. any ideas guys?

Re: Splunk on-demand indexing of a file

Ayn — Wed, 06 Feb 2013 09:41:21 GMT

I would say no to your question - there might in certain situations be some kind of hack that makes you do this but in essence this is not how Splunk was meant to work.