topic Re: How to disable an alert on an event within "x" time of another event? in Splunk Search

How to disable an alert on an event within "x" time of another event?

WoolarCJ — Wed, 01 Feb 2017 12:02:57 GMT

Hello,

We have 2 searches, one gets us a result that says something along the lines of "this product was removed". The other search gets a result that says "This product was installed". We are wondering if there is a way to not be alerted when a product was removed and the same product was reinstalled within say a 5 minute window. Any help and direction is appreciated.

Re: How to disable an alert on an event within "x" time of another event?

lguinn2 — Wed, 01 Feb 2017 21:39:08 GMT

Yes, it is entirely possible. However, the community will need to see both searches if you want us to help you write the search for the alert. (You can obscure the data, ips, values and even field names if you need to. )

Re: How to disable an alert on an event within "x" time of another event?

DalJeanis — Wed, 01 Feb 2017 22:50:56 GMT

Ten months more experience, and here's how I'd do it today...

 ( earliest =-65m@m  latest=@5m index=foo "was removed") OR
 ( earliest =-60m@m  latest=@0m index=foo "was installed")
 | rex "was (?<Status>removed|installed)"
 | stats count(eval(case(Status="removed",1))) as countRem, 
     count(eval(case(Status="installed",1))) as countInst, 
     max(eval(case(Status="removed",_time))) as maxRemTime, 
     min(eval(case(Status="installed",_time))) as minInstTime, 
     max(eval(case(Status="installed",_time))) as maxInstTime 
     by host product
 | eval AlertText = case(countInst==0, "Not Reinstalled after Removal",
     countRem==0, "Installed Okay",
     maxInstTime < maxRemTime, "Not Reinstalled after Removal",
     minInstTime > maxRemTime + 300, "Delayed Reinstall more than 5 minutes"
     true(), "Installed Okay")
 | search AlertText!="Installed Okay"

This code left here as a warning to others of how sloppy I used to be...

Here's a largely pseudocoded version of one way you could do it. I assumed you were running the search hourly, but the only significant thing about the times is that the Removed search time period needs to start 5 min before the Installed search time period starts, and end 5 min before the Installed search does.

earliest =-65m@m  latest=@5m [search "was removed] 
| ...extract the host and product... 
| stats count as countRem, max(_time) as maxRemTime by host product

| append
    [earliest =-60m@m  latest=@0m [search "was installed"] 
    | ...extract the host and product... 
    | stats count as countInst, min(_time) as minInstTime, max(_time) as maxInstTime by host product
    ] 

| stats sum(countRem) as countRem,  sum(countInst) as countInst, max(maxRemTime) as maxRemTime,
    min(minInstTime) as minInstTime, max(maxInstTime) as maxInstTime by host product
| eval AlertText = case(countInst==0, "Not Reinstalled after Removal",
    countRem==0, "Installed Okay",
    maxInstTime < maxRemTime, "Not Reinstalled after Removal",
    minInstTime > maxRemTime + 300, "Delayed Reinstall more than 5 minutes"
    true(), "Installed Okay")
| search AlertText!="Installed Okay"

Re: How to disable an alert on an event within "x" time of another event?

WoolarCJ — Tue, 29 Sep 2020 12:39:27 GMT

Sure thing,

Search 1: This gets us the uninstall event
index=wineventlog sourcetype="WinEventLog:Application" host!=DEV* host!=MIR* host!=TST host!=TMP* (EventCode=11724 OR EventCode=1034)
| rex field=Message "(?s)Product: (?.) --|(?s)Product\sName:\s(?.).\sProduct\sVersion"
| rex field=Message "--\s(?.).|(?.).\sProduct\sName:"
| rex field=Message "error\sstatus:\s(?\d*)."
| eval Software=coalesce(product_name,product_names)
| eval Result=coalesce(action,actions)
| lookup AccountNameSid objectSid as Sid
| eval user=if(isnull(sAMAccountName), "User Not Defined", "")
| eval "Uninstalled By"=coalesce(sAMAccountName,user)
| search Software="[PRODUCT] Result="remov" error_code="0"
| fields _time host Software Result "Uninstalled By" Message
| table _time host Software Result "Uninstalled By"
| rename _time as "Date/Time" host as Host
| convert timeformat="%Y/%m/%d %H:%M:%S" ctime("Date/Time")

Search 2: This gets us the Install event
index=wineventlog sourcetype="WinEventLog:Application" host!=DEV* host!=MIR* host!=TST host!=TMP* (EventCode=11707 OR EventCode=1033)
| rex field=Message "(?s)Product: (?.) --|(?s)Product\sName:\s(?.).\sProduct\sVersion"
| rex field=Message "--\s(?.).|(?.).\sProduct\sName:"
| rex field=Message "error\sstatus:\s(?\d*)."
| eval Software=coalesce(product_name,product_names)
| eval Result=coalesce(action,actions)
| lookup AccountNameSid objectSid as Sid
| eval user=if(isnull(sAMAccountName), "User Not Defined", "")
| eval "Installed By"=coalesce(sAMAccountName,user)
| search Software="[PRODUCT]*"
| fields _time host Software Result "Installed By" Message
| table _time host Software Result "Installed By"
| rename _time as "Date/Time" host as Host
| convert timeformat="%Y/%m/%d %H:%M:%S" ctime("Date/Time")

Please feel free to modify the searches, there is quite a few things in there to make it more friendly to the people who would read this. Thanks again for the help.

Re: How to disable an alert on an event within "x" time of another event?

raoul — Tue, 28 Nov 2017 17:16:31 GMT

We have done a similar thing using the Splunk transaction command.

You have a base search that returns the "removed" as a a startswith and the "installed" as an endswith option on the transaction command. Then set a maxspan=5m and then finally filter on results where the eventcount field that is added != 2.