How to create alert when no set message for X minutes from particular source type on host

[akashsawant]

Hello, 

We have a PowerShell script job ( xyz.ps1 ) run on all hosts every 10 minutes and when it starts write message in to EV Application log as "Beginning of xyz.ps1 Execution " , We found sometime that xyz.ps1 gets stuck into weird state and we didnt see message in last 60 minutes for some hosts. I was able to create alert where i get list of hosts which shows that message. But I am exactly looking for :

I want to set an alert in splunk which will report host name where we dont see "Beginning of xyz.ps1 Execution" message in last 60 minutes , So that I'll get to know these hosts where script didnt execute well.

search:   index= ABC source="xyz.ps1" host = WWW-*  "Beginning of xyz.ps1 Execution" | table _time host | dedup host | eval age=now()-_time | where age > 60

Is above search is correct ? Thanks for your suggestions

[akashsawant · Accepted solution]

Thanks @isoutamo for your time and suggestion. I made it work using following search:

index= ABC source="xyz.ps1" host = WWW-* Message="Beginning of xyz.ps1 Execution"
| dedup host
| eval age=now()-_time
| where age > 3600
| table _time host age

Note: 3600 means 60 minutes ( 60 x 60 )

[isoutamo]

Hi
The issue with this query is that it expects that there is at least one event on log within your search time. But if there haven’t that event then it didn’t know that there is a missing host….

[isoutamo]

Hi

maybe this will be more efficient than your?

index= ABC source="xyz.ps1" host = WWW-* "Beginning of xyz.ps1 Execution" 
| stats last(_time) as _time by host 
| append 
    [| makeresults 
    | eval host="WWW-foobar" 
    | eval _time=0] 
| eval age=now()-_time 
| where age > 60*60

If you have only couple of hosts you could add those inside append block host part and if there are many or names will changed often then replace that with lookup. Just store those host into lookup from where you are reading host + _time as last seen.

r. Ismo