Splunk Search

Is it possible to bundle multiple searches together?

balidani
Explorer

Hello!

I'm trying to run many queries on a log every day. Is it possible to bundle these searches together, so Splunk doesn't have to iterate over the whole log every time?

I tried searching for an answere here and in the documentation, but I didn't manage to find anything.
Thanks in advance!

0 Karma
1 Solution

lguinn2
Legend

Okay, here is how I would do some of these

sourcetype="nginx_log" earliest=-1d@d latest=@d
| eval urlGroup = ""
| eval urlGroup = case(match(url,"^/ws/2/label/"),"/ws/2/label/",
                       match(url,"^/ws/2/artist/"),"/ws/2/artist/", 
                       match(url,"^/ws/2/release-group/"),"/ws/2/release-group/", 
                       match(url,"^/ws/2/release/"),"/ws/2/release/",
                       match(url,"^/ws/2/recording/"),"/ws/2/recording/",
                       match(url,"^/ws/2/work/"),"/ws/2/work/")
| top 50 inc by urlGroup

Would do the first 7 searches as a single search. The eighth search can be, in and of itself, dramatically simplified as follows:

sourcetype="nginx_log"  query="*" earliest=-1d@d latest=@d
| fields url
| eval name = ""
| eval name = case(match(url,"^/ws/1/artist/","Artists",
                   match(url,"^/ws/1/track/"),"Tracks", 
                   match(url,"^/ws/1/release-group/"),"Release-group", 
                   match(url,"^/ws/1/release/"),"Releases",
                   match(url,"^/ws/1/label/"),"Labels")
| stats count by name

This should be more efficient. The remaining searches could follow this second pattern.

Try it and see what you think.

View solution in original post

lguinn2
Legend

Okay, here is how I would do some of these

sourcetype="nginx_log" earliest=-1d@d latest=@d
| eval urlGroup = ""
| eval urlGroup = case(match(url,"^/ws/2/label/"),"/ws/2/label/",
                       match(url,"^/ws/2/artist/"),"/ws/2/artist/", 
                       match(url,"^/ws/2/release-group/"),"/ws/2/release-group/", 
                       match(url,"^/ws/2/release/"),"/ws/2/release/",
                       match(url,"^/ws/2/recording/"),"/ws/2/recording/",
                       match(url,"^/ws/2/work/"),"/ws/2/work/")
| top 50 inc by urlGroup

Would do the first 7 searches as a single search. The eighth search can be, in and of itself, dramatically simplified as follows:

sourcetype="nginx_log"  query="*" earliest=-1d@d latest=@d
| fields url
| eval name = ""
| eval name = case(match(url,"^/ws/1/artist/","Artists",
                   match(url,"^/ws/1/track/"),"Tracks", 
                   match(url,"^/ws/1/release-group/"),"Release-group", 
                   match(url,"^/ws/1/release/"),"Releases",
                   match(url,"^/ws/1/label/"),"Labels")
| stats count by name

This should be more efficient. The remaining searches could follow this second pattern.

Try it and see what you think.

balidani
Explorer

Thank you! Querying takes a significantly shorter amount of time now.

0 Karma

balidani
Explorer

Thank you for your reply! Here is a sample of the queries I'm trying to run:

https://gist.github.com/3499469

The log is a basic nginx log. You can see that most of the queries contain 'top', or many 'count's.

0 Karma

lguinn2
Legend

Maybe, it depends on the searches. If you give us a sample (2 or 3) of the searches, and a few lines of the log... we might be able to come up with some ideas for you.

I find that it is often possible to reduce the number of searches, even when you can't bundle all of them together.

Get Updates on the Splunk Community!

Splunk Classroom Chronicles: Training Tales and Testimonials (Episode 3)

Welcome back to Splunk Classroom Chronicles, our ongoing blog series that pulls back the curtain on Splunk ...

Operationalizing TDIR: Building a More Resilient, Scalable SOC

Optimizing SOC workflows with a unified, risk-based approach to Threat Detection, Investigation, and Response ...

Almost Too Eventful Assurance: Part 1

Modern IT and Network teams still struggle with too many alerts and isolating issues before they are notified. ...