We have a PowerShell script job ( xyz.ps1 ) run on all hosts every 10 minutes and when it starts write message in to EV Application log as "Beginning of xyz.ps1 Execution " , We found sometime that xyz.ps1 gets stuck into weird state and we didnt see message in last 60 minutes for some hosts. I was able to create alert where i get list of hosts which shows that message. But I am exactly looking for :
I want to set an alert in splunk which will report host name where we dont see "Beginning of xyz.ps1 Execution" message in last 60 minutes , So that I'll get to know these hosts where script didnt execute well.
search: index= ABC source="xyz.ps1" host = WWW-* "Beginning of xyz.ps1 Execution" | table _time host | dedup host | eval age=now()-_time | where age > 60
Is above search is correct ? Thanks for your suggestions
index= ABC source="xyz.ps1" host = WWW-* "Beginning of xyz.ps1 Execution"
| stats last(_time) as _time by host
| eval host="WWW-foobar"
| eval _time=0]
| eval age=now()-_time
| where age > 60*60
If you have only couple of hosts you could add those inside append block host part and if there are many or names will changed often then replace that with lookup. Just store those host into lookup from where you are reading host + _time as last seen.