I currently have a FIX log file which generates HUGE amounts of data every day. With my current license its impossible to index all that data.
Is it possible to get parts of the log file on-demand and then index and query them as and when needed? Is it possible to achieve this via a script invoked from a Splunk query?
or basically any way to directly query data without indexing it in Splunk?
Also, see routing and filtering data, specifically to a null queue: http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Routeandfilterdatad
Yes, you can use scripted input (http://docs.splunk.com/Documentation/Splunk/5.0.1/AdvancedDev/ScriptSetup).
I would say no to your question - there might in certain situations be some kind of hack that makes you do this but in essence this is not how Splunk was meant to work.
"Currently, it is not supported by Splunk"-- Not sure whether this command works anymore. any ideas guys?
Possibly, but only if the data to index is located on the Splunk search head (or is reachable via a script on the search head). See http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Runshellscript . This will run a script on the search head.
ok sounds good in terms of permissions but as you mentioned this can be scheduled to run via cron or task scheduler. What I need is to run it on demand anytime and gat her data only from a particular time period(which I think can be done via arguements to the script). Bu thow to invoke the script on demand rather than on a schedule?
Scripted inputs run under the context of the Splunk user. If that doesn't work for your specific needs, you could create a script that is run via Windows task scheduler (which will let you select the user to run the script as). The script just needs to do what ever parsing of the original log file you want, and then output it to a new file that you can have Splunk monitor/index.
scripted input don't think would fit the case as the script should be invokable via non admin users as well, which I am not sure can be done