How to NOT alert on events that are caused by a reboot

[cmahan]

What is the best way to not alert on events that are caused by a reboot. For example - I am alerting on a WMI:Service Status=Stopped for a particular service. I don't want to get an alert if the server was rebooting and that is why the service was in the stopped state. Might be a bad example as the result might not be what I expect, but I need to be able to accomplish the general concept for other alerts. I think I am looking for a subsearch? but not sure how to write it..?

blah blah event | subsearch NOT "i just rebooted" within last 10 minutes

something along those lines

Then if the event or state is detected, but also a reboot event was detected within the last 10 minutes, then no alert is triggered. If there was no reboot, the alert IS triggered.

[MuS]

Hi cmahan,

take a look at this run everywhere command:

index=_internal sourcetype=splunkd series=splunkd earliest=-11min@min 
| bucket _time span=10min 
| search NOT [ search index=_audit action=search ] 
| timechart span=1min count

it searches for events in the _internal index over the last 11 minutes, creates buckets of 10 minutes span and searches within this time span if there was no match for action=search in index=_audit. Yes this is a non-sense use case but it will help you understand your alert 😉

Based on this and your comments, try something like:

sourcetype="WMI:Service" Name=slinksc State=Stopped earliest=-11min@min 
| bucket _time span=10min 
| search NOT [ search  EventCode=1074 ]

I removed the map command, because it is just too messy.

cheers, MuS

[cmahan]

Thanks! I will give that a try.

[cmahan]

No dice. Tried both of these without success.

sourcetype="WMI:Service" Name=slinksc State=Stopped | search NOT [ search earliest>=-10min@min EventCode=1074 ]

sourcetype="WMI:Service" Name=slinksc State=Stopped | eval time=_time | map search="search NOT EventCode=1074 earliest>=($time$ - 600 ) latest<=($time$)"

[MuS]

okay, twice my bad....

just learned this answer won't work for the first one using subsearch!! Why? Because Splunk runs the inner search using the same info from the timerange picker as the outer search.
And the map is also wrong, because the NOT will not work 😞 ... I'll work on a updated version....

[MuS]

update ping, see the updated answer

[cmahan]

Tried it with great hopes! but alas, the event still shows up. It seems like its ignoring the subsearch still. Are we sure that syntax is conveying "don't return anything if you find results matching the subsearch along with the main search, even if the main search has results?" I'm actually trying with a different service now. Here is what i have tried - only difference between the 2 is the way I defined the State to include all states besides Running:

sourcetype="WMI:Service" Name="ds_agent" State="Stopped" OR State="Start*" earliest=-11min@min | bucket _time span=10min | search NOT [ search EventCode=1074 ]

and

sourcetype="WMI:Service" Name="ds_agent" NOT State=Running earliest=-11min@min | bucket _time span=10min | search NOT [ search EventCode=1074 ]