Splunk Search

Stats / counts in a subsearch for day-over-day delta alerting


I'm trying to set up an alert based on day-over-day vs. last week for a 5m count. For instance, if certain iis events delta over a certain percentage compared to last week's count.

I can graph the three (Today, Last_Week, and Difference) using this:

index=iis latest=now earliest=-5m sc_status=200 | timechart count as Today | appendcols [search index=iis latest=-10080m earliest=-10085m sc_status=200 | timechart count as Last_Week ] | eval Difference = Today - Last_Week

My question is, how do I turn that into a search I can run every 5 minutes that returns only the "Difference" in percentage form so I can set up an alert if it changes more than an acceptable amount? I've tried using stats, but it seems nesting it in a subsearch creates an issue.

index=iis latest=now earliest=-24h sc_status=200 | stats count as Today | 
appendcols [search index=iis latest=-168h earliest=-144h sc_status=200 | stats count as Last_Week ] | 
eval Difference = Today - Last_Week | fields + Today Last_Week Difference

When you create the alert, use a custom trigger where Difference > some amount. Set the schedule to run this search every 5 minutes.

You can definitely use stats in a subsearch. In fact, I think that what you want is stats, not timechart. But I am not clear on what you mean by day-over-day vs. last week. Do you want to compare today's count with the same day of last week?

Finally, summary indexing might be helpful. Keep an hour-by-hour summary count, and then your subsearch could run against the summary index VERY quickly.

Get Updates on the Splunk Community!

Splunk Observability Cloud | Customer Survey!

If you use Splunk Observability Cloud, we invite you to share your valuable insights with us through a brief ...

Happy CX Day, Splunk Community!

Happy CX Day, Splunk Community! CX stands for Customer Experience, and today, October 3rd, is CX Day — a ...

.conf23 | Get Your Cybersecurity Defense Analyst Certification in Vegas

We’re excited to announce a new Splunk certification exam being released at .conf23! If you’re going to Las ...