Splunk Search

Please Help Me Extend This Search

niall_munnelly
Path Finder

I'm trying to automate sending a "clear" Splunk alert by comparing results from a previous search with the current one. This search works:

source="/path/to/error.log" earliest=-10m@m latest=-5m@m "file does not exist" | eval results="Old" | append [search source="/path/to/error.log" earliest=-5m@m latest=now "file does not exist" | eval results="New"] | stats sum(eval(if(match(results,"Old"),1,0))) as Old sum(eval(if(match(results,"New"),1,0))) as New

(I'm not alerting for 404s - I just needed some data!)

This gives me a nice, two-column view of the Old and New results and, for the alert trigger, the custom condition is "search Old>0 AND New=0."

The trouble is that I don't want to monitor a single log - I want to monitor a sourcetype, and to table these results by host and source. The eval statements are the only way I can think of to keep the old and new values distinct - "stats count by host, source" just dumps everything into a single value. I'd like to see something like:

host, source, old, new

webserver1, web1_error.log, 15, 3

webserver2, web2_error.log, 24, 0

Am I on the right track? I'd be grateful for any nudges in the right direction, because I've hit a wall.

Tags (3)
0 Karma
1 Solution

lguinn2
Legend

Try this

sourcetype=xyz earliest=-10m@m latest=@m yourothercriteria
| eval timespan=if(_time < (now()-300),"Old","New")
| stats count(eval(timespan="Old")) as Old  count(eval(timespan="New")) as New by host source

This will also be more efficient than the subsearch technique. now() is the time that the search started running, so now()-300 is 5 minutes before that.

View solution in original post

lguinn2
Legend

Try this

sourcetype=xyz earliest=-10m@m latest=@m yourothercriteria
| eval timespan=if(_time < (now()-300),"Old","New")
| stats count(eval(timespan="Old")) as Old  count(eval(timespan="New")) as New by host source

This will also be more efficient than the subsearch technique. now() is the time that the search started running, so now()-300 is 5 minutes before that.

niall_munnelly
Path Finder

So much more elegant than mine! This is excellent - I can see it working already. Thanks!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...