Alerting

How to set up an alert when an application goes down between a certain time (7 AM till 10 PM)?

arjitgoswami
Explorer

Hi All, I am in a process of setting up real-time application log monitoring tool using Splunk which notifies users whenever the application goes down (by checking whether the Splunk is receiving logs or not). I need the system to generate alert only when it sees there are no incoming logs from 7 AM till 10 PM on a weekday as our system is only up during a certain time in a day. How can this be achieved?

Thanks for your help!

Regards,
Arjit.

Tags (1)
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi arjitgoswami,
there are different ways to limit your check only to some hours and days:

index=your_index app_process

and set an alert with cron

0 7-22 * * 1-5

that triggers when there isn't any result.

Or if you have an error message

index=your_index error_message (date_hour>6 date_hour<23) NOT (date_wday="Sunday" OR date_wday="Saturday")

that triggers when there's a result

Bye.
Giuseppe

View solution in original post

arjitgoswami
Explorer

@cusello @somesoni2 @skoelpin : Thanks a lot for your help ! Appreciate it!!

0 Karma

skoelpin
SplunkTrust
SplunkTrust

Feel free to upvote/accept an answer if it helped 🙂

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi arjitgoswami,
there are different ways to limit your check only to some hours and days:

index=your_index app_process

and set an alert with cron

0 7-22 * * 1-5

that triggers when there isn't any result.

Or if you have an error message

index=your_index error_message (date_hour>6 date_hour<23) NOT (date_wday="Sunday" OR date_wday="Saturday")

that triggers when there's a result

Bye.
Giuseppe

somesoni2
SplunkTrust
SplunkTrust

Using cron schedule to run on specific period would be better as it avoids unwanted execution (others also will not do any harm but since your alert condition would be negative, you'd need a more complex alert search if you use filters inline in search). Also, (ignore if you were not planning) instead of using real-time search alert, use a rather frequent alerts e.g. say every 5 mins.

0 Karma

skoelpin
SplunkTrust
SplunkTrust

It would have to look something like this

index=_internal
| eval hour=strftime(_time,"%H")
| eval alert=if(hour>07 AND hour<22,"ALERT","NORMAL")

You will need to add your logic to this, but this gives you an idea of how it works. We are extracting the hour from _time and its checking to see if your condition is within that timeframe from the logic on line 3. You should then save it as an alert and select Custom then add search alert="ALERT"

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...