topic Re: Is it possible to bundle multiple searches together? in Splunk Search

Is it possible to bundle multiple searches together?

balidani — Thu, 23 Aug 2012 18:33:31 GMT

Hello!

I'm trying to run many queries on a log every day. Is it possible to bundle these searches together, so Splunk doesn't have to iterate over the whole log every time?

I tried searching for an answere here and in the documentation, but I didn't manage to find anything.
Thanks in advance!

Re: Is it possible to bundle multiple searches together?

lguinn2 — Mon, 27 Aug 2012 20:13:29 GMT

Maybe, it depends on the searches. If you give us a sample (2 or 3) of the searches, and a few lines of the log... we might be able to come up with some ideas for you.

I find that it is often possible to reduce the number of searches, even when you can't bundle all of them together.

Re: Is it possible to bundle multiple searches together?

balidani — Tue, 28 Aug 2012 16:03:23 GMT

Thank you for your reply! Here is a sample of the queries I'm trying to run:

https://gist.github.com/3499469

The log is a basic nginx log. You can see that most of the queries contain 'top', or many 'count's.

Re: Is it possible to bundle multiple searches together?

lguinn2 — Wed, 29 Aug 2012 21:50:53 GMT

Okay, here is how I would do some of these

sourcetype="nginx_log" earliest=-1d@d latest=@d
| eval urlGroup = ""
| eval urlGroup = case(match(url,"^/ws/2/label/"),"/ws/2/label/",
                       match(url,"^/ws/2/artist/"),"/ws/2/artist/", 
                       match(url,"^/ws/2/release-group/"),"/ws/2/release-group/", 
                       match(url,"^/ws/2/release/"),"/ws/2/release/",
                       match(url,"^/ws/2/recording/"),"/ws/2/recording/",
                       match(url,"^/ws/2/work/"),"/ws/2/work/")
| top 50 inc by urlGroup

Would do the first 7 searches as a single search. The eighth search can be, in and of itself, dramatically simplified as follows:

sourcetype="nginx_log"  query="*" earliest=-1d@d latest=@d
| fields url
| eval name = ""
| eval name = case(match(url,"^/ws/1/artist/","Artists",
                   match(url,"^/ws/1/track/"),"Tracks", 
                   match(url,"^/ws/1/release-group/"),"Release-group", 
                   match(url,"^/ws/1/release/"),"Releases",
                   match(url,"^/ws/1/label/"),"Labels")
| stats count by name

This should be more efficient. The remaining searches could follow this second pattern.

Try it and see what you think.

Re: Is it possible to bundle multiple searches together?

balidani — Sun, 09 Sep 2012 23:17:07 GMT

Thank you! Querying takes a significantly shorter amount of time now.