Splunk Search

Is it possible to bundle multiple searches together?

balidani
Explorer

Hello!

I'm trying to run many queries on a log every day. Is it possible to bundle these searches together, so Splunk doesn't have to iterate over the whole log every time?

I tried searching for an answere here and in the documentation, but I didn't manage to find anything.
Thanks in advance!

0 Karma
1 Solution

lguinn2
Legend

Okay, here is how I would do some of these

sourcetype="nginx_log" earliest=-1d@d latest=@d
| eval urlGroup = ""
| eval urlGroup = case(match(url,"^/ws/2/label/"),"/ws/2/label/",
                       match(url,"^/ws/2/artist/"),"/ws/2/artist/", 
                       match(url,"^/ws/2/release-group/"),"/ws/2/release-group/", 
                       match(url,"^/ws/2/release/"),"/ws/2/release/",
                       match(url,"^/ws/2/recording/"),"/ws/2/recording/",
                       match(url,"^/ws/2/work/"),"/ws/2/work/")
| top 50 inc by urlGroup

Would do the first 7 searches as a single search. The eighth search can be, in and of itself, dramatically simplified as follows:

sourcetype="nginx_log"  query="*" earliest=-1d@d latest=@d
| fields url
| eval name = ""
| eval name = case(match(url,"^/ws/1/artist/","Artists",
                   match(url,"^/ws/1/track/"),"Tracks", 
                   match(url,"^/ws/1/release-group/"),"Release-group", 
                   match(url,"^/ws/1/release/"),"Releases",
                   match(url,"^/ws/1/label/"),"Labels")
| stats count by name

This should be more efficient. The remaining searches could follow this second pattern.

Try it and see what you think.

View solution in original post

lguinn2
Legend

Okay, here is how I would do some of these

sourcetype="nginx_log" earliest=-1d@d latest=@d
| eval urlGroup = ""
| eval urlGroup = case(match(url,"^/ws/2/label/"),"/ws/2/label/",
                       match(url,"^/ws/2/artist/"),"/ws/2/artist/", 
                       match(url,"^/ws/2/release-group/"),"/ws/2/release-group/", 
                       match(url,"^/ws/2/release/"),"/ws/2/release/",
                       match(url,"^/ws/2/recording/"),"/ws/2/recording/",
                       match(url,"^/ws/2/work/"),"/ws/2/work/")
| top 50 inc by urlGroup

Would do the first 7 searches as a single search. The eighth search can be, in and of itself, dramatically simplified as follows:

sourcetype="nginx_log"  query="*" earliest=-1d@d latest=@d
| fields url
| eval name = ""
| eval name = case(match(url,"^/ws/1/artist/","Artists",
                   match(url,"^/ws/1/track/"),"Tracks", 
                   match(url,"^/ws/1/release-group/"),"Release-group", 
                   match(url,"^/ws/1/release/"),"Releases",
                   match(url,"^/ws/1/label/"),"Labels")
| stats count by name

This should be more efficient. The remaining searches could follow this second pattern.

Try it and see what you think.

View solution in original post

balidani
Explorer

Thank you! Querying takes a significantly shorter amount of time now.

0 Karma

balidani
Explorer

Thank you for your reply! Here is a sample of the queries I'm trying to run:

https://gist.github.com/3499469

The log is a basic nginx log. You can see that most of the queries contain 'top', or many 'count's.

0 Karma

lguinn2
Legend

Maybe, it depends on the searches. If you give us a sample (2 or 3) of the searches, and a few lines of the log... we might be able to come up with some ideas for you.

I find that it is often possible to reduce the number of searches, even when you can't bundle all of them together.

Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!