Splunk Search

Stats / counts in a subsearch for day-over-day delta alerting

cmaier
Explorer

I'm trying to set up an alert based on day-over-day vs. last week for a 5m count. For instance, if certain iis events delta over a certain percentage compared to last week's count.

I can graph the three (Today, Last_Week, and Difference) using this:

index=iis latest=now earliest=-5m sc_status=200 | timechart count as Today | appendcols [search index=iis latest=-10080m earliest=-10085m sc_status=200 | timechart count as Last_Week ] | eval Difference = Today - Last_Week

My question is, how do I turn that into a search I can run every 5 minutes that returns only the "Difference" in percentage form so I can set up an alert if it changes more than an acceptable amount? I've tried using stats, but it seems nesting it in a subsearch creates an issue.

lguinn2
Legend
index=iis latest=now earliest=-24h sc_status=200 | stats count as Today | 
appendcols [search index=iis latest=-168h earliest=-144h sc_status=200 | stats count as Last_Week ] | 
eval Difference = Today - Last_Week | fields + Today Last_Week Difference

When you create the alert, use a custom trigger where Difference > some amount. Set the schedule to run this search every 5 minutes.

You can definitely use stats in a subsearch. In fact, I think that what you want is stats, not timechart. But I am not clear on what you mean by day-over-day vs. last week. Do you want to compare today's count with the same day of last week?

Finally, summary indexing might be helpful. Keep an hour-by-hour summary count, and then your subsearch could run against the summary index VERY quickly.

Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In December, the Splunk Threat Research Team had 1 release of new security content via the Enterprise Security ...

Why am I not seeing the finding in Splunk Enterprise Security Analyst Queue?

(This is the first of a series of 2 blogs). Splunk Enterprise Security is a fantastic tool that offers robust ...

Index This | What are the 12 Days of Splunk-mas?

December 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...