Alerting

How to set up an alert when an application goes down between a certain time (7 AM till 10 PM)?

arjitgoswami
New Member

Hi All, I am in a process of setting up real-time application log monitoring tool using Splunk which notifies users whenever the application goes down (by checking whether the Splunk is receiving logs or not). I need the system to generate alert only when it sees there are no incoming logs from 7 AM till 10 PM on a weekday as our system is only up during a certain time in a day. How can this be achieved?

Thanks for your help!

Regards,
Arjit.

Tags (1)
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi arjitgoswami,
there are different ways to limit your check only to some hours and days:

index=your_index app_process

and set an alert with cron

0 7-22 * * 1-5

that triggers when there isn't any result.

Or if you have an error message

index=your_index error_message (date_hour>6 date_hour<23) NOT (date_wday="Sunday" OR date_wday="Saturday")

that triggers when there's a result

Bye.
Giuseppe

View solution in original post

arjitgoswami
New Member

@cusello @somesoni2 @skoelpin : Thanks a lot for your help ! Appreciate it!!

0 Karma

skoelpin
SplunkTrust
SplunkTrust

Feel free to upvote/accept an answer if it helped 🙂

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi arjitgoswami,
there are different ways to limit your check only to some hours and days:

index=your_index app_process

and set an alert with cron

0 7-22 * * 1-5

that triggers when there isn't any result.

Or if you have an error message

index=your_index error_message (date_hour>6 date_hour<23) NOT (date_wday="Sunday" OR date_wday="Saturday")

that triggers when there's a result

Bye.
Giuseppe

View solution in original post

somesoni2
Revered Legend

Using cron schedule to run on specific period would be better as it avoids unwanted execution (others also will not do any harm but since your alert condition would be negative, you'd need a more complex alert search if you use filters inline in search). Also, (ignore if you were not planning) instead of using real-time search alert, use a rather frequent alerts e.g. say every 5 mins.

0 Karma

skoelpin
SplunkTrust
SplunkTrust

It would have to look something like this

index=_internal
| eval hour=strftime(_time,"%H")
| eval alert=if(hour>07 AND hour<22,"ALERT","NORMAL")

You will need to add your logic to this, but this gives you an idea of how it works. We are extracting the hour from _time and its checking to see if your condition is within that timeframe from the logic on line 3. You should then save it as an alert and select Custom then add search alert="ALERT"

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!