Splunk Search

Stats / counts in a subsearch for day-over-day delta alerting

cmaier
Explorer

I'm trying to set up an alert based on day-over-day vs. last week for a 5m count. For instance, if certain iis events delta over a certain percentage compared to last week's count.

I can graph the three (Today, Last_Week, and Difference) using this:

index=iis latest=now earliest=-5m sc_status=200 | timechart count as Today | appendcols [search index=iis latest=-10080m earliest=-10085m sc_status=200 | timechart count as Last_Week ] | eval Difference = Today - Last_Week

My question is, how do I turn that into a search I can run every 5 minutes that returns only the "Difference" in percentage form so I can set up an alert if it changes more than an acceptable amount? I've tried using stats, but it seems nesting it in a subsearch creates an issue.

lguinn2
Legend
index=iis latest=now earliest=-24h sc_status=200 | stats count as Today | 
appendcols [search index=iis latest=-168h earliest=-144h sc_status=200 | stats count as Last_Week ] | 
eval Difference = Today - Last_Week | fields + Today Last_Week Difference

When you create the alert, use a custom trigger where Difference > some amount. Set the schedule to run this search every 5 minutes.

You can definitely use stats in a subsearch. In fact, I think that what you want is stats, not timechart. But I am not clear on what you mean by day-over-day vs. last week. Do you want to compare today's count with the same day of last week?

Finally, summary indexing might be helpful. Keep an hour-by-hour summary count, and then your subsearch could run against the summary index VERY quickly.

Get Updates on the Splunk Community!

Webinar Recap | Revolutionizing IT Operations: The Transformative Power of AI and ML ...

The Transformative Power of AI and ML in Enhancing Observability   In the realm of IT operations, the ...

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...