How to set up an alert when an application goes down between a certain time (7 AM till 10 PM)?

[arjitgoswami]

Hi All, I am in a process of setting up real-time application log monitoring tool using Splunk which notifies users whenever the application goes down (by checking whether the Splunk is receiving logs or not). I need the system to generate alert only when it sees there are no incoming logs from 7 AM till 10 PM on a weekday as our system is only up during a certain time in a day. How can this be achieved?

Thanks for your help!

Regards,
Arjit.

[arjitgoswami]

@cusello @somesoni2 @skoelpin : Thanks a lot for your help ! Appreciate it!!

[skoelpin]

Feel free to upvote/accept an answer if it helped 🙂

[gcusello · Accepted solution]

Hi arjitgoswami,
there are different ways to limit your check only to some hours and days:

index=your_index app_process

and set an alert with cron

0 7-22 * * 1-5

that triggers when there isn't any result.

Or if you have an error message

index=your_index error_message (date_hour>6 date_hour<23) NOT (date_wday="Sunday" OR date_wday="Saturday")

that triggers when there's a result

Bye.
Giuseppe

[somesoni2]

Using cron schedule to run on specific period would be better as it avoids unwanted execution (others also will not do any harm but since your alert condition would be negative, you'd need a more complex alert search if you use filters inline in search). Also, (ignore if you were not planning) instead of using real-time search alert, use a rather frequent alerts e.g. say every 5 mins.

[skoelpin]

It would have to look something like this

index=_internal
| eval hour=strftime(_time,"%H")
| eval alert=if(hour>07 AND hour<22,"ALERT","NORMAL")

You will need to add your logic to this, but this gives you an idea of how it works. We are extracting the hour from _time and its checking to see if your condition is within that timeframe from the logic on line 3. You should then save it as an alert and select Custom then add search alert="ALERT"