Alerting

How to create alert when no set message for X minutes from particular source type on host

akashsawant
Explorer

Hello, 

We have a PowerShell script job ( xyz.ps1 ) run on all hosts every 10 minutes and when it starts write message in to EV Application log as "Beginning of xyz.ps1 Execution " , We found sometime that xyz.ps1 gets stuck into weird state and we didnt see message in last 60 minutes for some hosts. I was able to create alert where i get list of hosts which shows that message. But I am exactly looking for :

I want to set an alert in splunk which will report host name where we dont see "Beginning of xyz.ps1 Execution" message in last 60 minutes , So that I'll get to know these hosts where script didnt execute well.

search:   index= ABC source="xyz.ps1" host = WWW-*  "Beginning of xyz.ps1 Execution" | table _time host | dedup host | eval age=now()-_time | where age > 60

Is above search is correct ? Thanks for your suggestions

Labels (2)
0 Karma
1 Solution

akashsawant
Explorer

Thanks @isoutamo for your time and suggestion. I made it work using following search:

index= ABC source="xyz.ps1" host = WWW-* Message="Beginning of xyz.ps1 Execution"
| dedup host
| eval age=now()-_time
| where age > 3600
| table _time host age

 

Note: 3600 means 60 minutes ( 60 x 60 )

View solution in original post

0 Karma

akashsawant
Explorer

Thanks @isoutamo for your time and suggestion. I made it work using following search:

index= ABC source="xyz.ps1" host = WWW-* Message="Beginning of xyz.ps1 Execution"
| dedup host
| eval age=now()-_time
| where age > 3600
| table _time host age

 

Note: 3600 means 60 minutes ( 60 x 60 )

0 Karma

isoutamo
SplunkTrust
SplunkTrust
Hi
The issue with this query is that it expects that there is at least one event on log within your search time. But if there haven’t that event then it didn’t know that there is a missing host….
0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

maybe this will be more efficient than your?

index= ABC source="xyz.ps1" host = WWW-* "Beginning of xyz.ps1 Execution" 
| stats last(_time) as _time by host 
| append 
    [| makeresults 
    | eval host="WWW-foobar" 
    | eval _time=0] 
| eval age=now()-_time 
| where age > 60*60

If you have only couple of hosts you could add those inside append block host part and if there are many or names will changed often then replace that with lookup. Just store those host into lookup from where you are reading host + _time as last seen.

r. Ismo 

Get Updates on the Splunk Community!

New Splunk Observability innovations: Deeper visibility and smarter alerting to ...

You asked, we delivered. Splunk Observability Cloud has several new innovations giving you deeper visibility ...

Synthetic Monitoring: Not your Grandma’s Polyester! Tech Talk: DevOps Edition

Register today and join TekStream on Tuesday, February 28 at 11am PT/2pm ET for a demonstration of Splunk ...

Instrumenting Java Websocket Messaging

Instrumenting Java Websocket MessagingThis article is a code-based discussion of passing OpenTelemetry trace ...