What is the best way to not alert on events that are caused by a reboot. For example - I am alerting on a WMI:Service Status=Stopped for a particular service. I don't want to get an alert if the server was rebooting and that is why the service was in the stopped state. Might be a bad example as the result might not be what I expect, but I need to be able to accomplish the general concept for other alerts. I think I am looking for a subsearch? but not sure how to write it..?
blah blah event | subsearch NOT "i just rebooted" within last 10 minutes
something along those lines
Then if the event or state is detected, but also a reboot event was detected within the last 10 minutes, then no alert is triggered. If there was no reboot, the alert IS triggered.
take a look at this run everywhere command:
index=_internal sourcetype=splunkd series=splunkd earliest=-11min@min | bucket _time span=10min | search NOT [ search index=_audit action=search ] | timechart span=1min count
it searches for events in the
_internal index over the last 11 minutes, creates buckets of 10 minutes span and searches within this time span if there was no match for
index=_audit. Yes this is a non-sense use case but it will help you understand your alert 😉
Based on this and your comments, try something like:
sourcetype="WMI:Service" Name=slinksc State=Stopped earliest=-11min@min | bucket _time span=10min | search NOT [ search EventCode=1074 ]
I removed the
map command, because it is just too messy.
No dice. Tried both of these without success.
sourcetype="WMI:Service" Name=slinksc State=Stopped | search NOT [ search earliest>=-10min@min EventCode=1074 ] sourcetype="WMI:Service" Name=slinksc State=Stopped | eval time=_time | map search="search NOT EventCode=1074 earliest>=($time$ - 600 ) latest<=($time$)"
okay, twice my bad....
just learned this answer won't work for the first one using subsearch!! Why? Because Splunk runs the inner search using the same info from the timerange picker as the outer search.
map is also wrong, because the
NOT will not work 😞 ... I'll work on a updated version....
Tried it with great hopes! but alas, the event still shows up. It seems like its ignoring the subsearch still. Are we sure that syntax is conveying "don't return anything if you find results matching the subsearch along with the main search, even if the main search has results?" I'm actually trying with a different service now. Here is what i have tried - only difference between the 2 is the way I defined the State to include all states besides Running:
sourcetype="WMI:Service" Name="ds_agent" State="Stopped" OR State="Start*" earliest=-11min@min | bucket _time span=10min | search NOT [ search EventCode=1074 ]
sourcetype="WMI:Service" Name="ds_agent" NOT State=Running earliest=-11min@min | bucket _time span=10min | search NOT [ search EventCode=1074 ]