Hi All, I am in a process of setting up real-time application log monitoring tool using Splunk which notifies users whenever the application goes down (by checking whether the Splunk is receiving logs or not). I need the system to generate alert only when it sees there are no incoming logs from 7 AM till 10 PM on a weekday as our system is only up during a certain time in a day. How can this be achieved?
Thanks for your help!
Regards,
Arjit.
Hi arjitgoswami,
there are different ways to limit your check only to some hours and days:
index=your_index app_process
and set an alert with cron
0 7-22 * * 1-5
that triggers when there isn't any result.
Or if you have an error message
index=your_index error_message (date_hour>6 date_hour<23) NOT (date_wday="Sunday" OR date_wday="Saturday")
that triggers when there's a result
Bye.
Giuseppe
@cusello @somesoni2 @skoelpin : Thanks a lot for your help ! Appreciate it!!
Feel free to upvote/accept an answer if it helped 🙂
Hi arjitgoswami,
there are different ways to limit your check only to some hours and days:
index=your_index app_process
and set an alert with cron
0 7-22 * * 1-5
that triggers when there isn't any result.
Or if you have an error message
index=your_index error_message (date_hour>6 date_hour<23) NOT (date_wday="Sunday" OR date_wday="Saturday")
that triggers when there's a result
Bye.
Giuseppe
Using cron schedule to run on specific period would be better as it avoids unwanted execution (others also will not do any harm but since your alert condition would be negative, you'd need a more complex alert search if you use filters inline in search). Also, (ignore if you were not planning) instead of using real-time search alert, use a rather frequent alerts e.g. say every 5 mins.
It would have to look something like this
index=_internal
| eval hour=strftime(_time,"%H")
| eval alert=if(hour>07 AND hour<22,"ALERT","NORMAL")
You will need to add your logic to this, but this gives you an idea of how it works. We are extracting the hour from _time
and its checking to see if your condition is within that timeframe from the logic on line 3. You should then save it as an alert and select Custom
then add search alert="ALERT"