Alerting

How to create alert when no set message for X minutes from particular source type on host

akashsawant
Explorer

Hello, 

We have a PowerShell script job ( xyz.ps1 ) run on all hosts every 10 minutes and when it starts write message in to EV Application log as "Beginning of xyz.ps1 Execution " , We found sometime that xyz.ps1 gets stuck into weird state and we didnt see message in last 60 minutes for some hosts. I was able to create alert where i get list of hosts which shows that message. But I am exactly looking for :

I want to set an alert in splunk which will report host name where we dont see "Beginning of xyz.ps1 Execution" message in last 60 minutes , So that I'll get to know these hosts where script didnt execute well.

search:   index= ABC source="xyz.ps1" host = WWW-*  "Beginning of xyz.ps1 Execution" | table _time host | dedup host | eval age=now()-_time | where age > 60

Is above search is correct ? Thanks for your suggestions

Labels (2)
0 Karma
1 Solution

akashsawant
Explorer

Thanks @isoutamo for your time and suggestion. I made it work using following search:

index= ABC source="xyz.ps1" host = WWW-* Message="Beginning of xyz.ps1 Execution"
| dedup host
| eval age=now()-_time
| where age > 3600
| table _time host age

 

Note: 3600 means 60 minutes ( 60 x 60 )

View solution in original post

0 Karma

akashsawant
Explorer

Thanks @isoutamo for your time and suggestion. I made it work using following search:

index= ABC source="xyz.ps1" host = WWW-* Message="Beginning of xyz.ps1 Execution"
| dedup host
| eval age=now()-_time
| where age > 3600
| table _time host age

 

Note: 3600 means 60 minutes ( 60 x 60 )

0 Karma

isoutamo
SplunkTrust
SplunkTrust
Hi
The issue with this query is that it expects that there is at least one event on log within your search time. But if there haven’t that event then it didn’t know that there is a missing host….
0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

maybe this will be more efficient than your?

index= ABC source="xyz.ps1" host = WWW-* "Beginning of xyz.ps1 Execution" 
| stats last(_time) as _time by host 
| append 
    [| makeresults 
    | eval host="WWW-foobar" 
    | eval _time=0] 
| eval age=now()-_time 
| where age > 60*60

If you have only couple of hosts you could add those inside append block host part and if there are many or names will changed often then replace that with lookup. Just store those host into lookup from where you are reading host + _time as last seen.

r. Ismo 

Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...