Alerting

alert in splunk

loveforsplunk
Explorer

start time = -2h@h Finish Time = now.

Can I use this in an alert in Splunk.. which will run my search every minute for the last 2 hours and will throw an alert if gets any result.

Tags (1)
0 Karma
1 Solution

dineshraj9
Builder

Try running the search once and check how much time it takes. If you have huge volume of data being searched then ensure that there are not 2 concurrent execution. Setting up every minute for 2hr window would work fine, but if you are looking for a keyword in the raw events, it is advisable that the search window and frequency remain same.(search -2m@m to -1m@m every minute).

Please check for timezone issue with the logs and fix them if any.

View solution in original post

0 Karma

dineshraj9
Builder

Try running the search once and check how much time it takes. If you have huge volume of data being searched then ensure that there are not 2 concurrent execution. Setting up every minute for 2hr window would work fine, but if you are looking for a keyword in the raw events, it is advisable that the search window and frequency remain same.(search -2m@m to -1m@m every minute).

Please check for timezone issue with the logs and fix them if any.

0 Karma

loveforsplunk
Explorer

I have one question here.. I am getting the alert when I kept the search as -2h@h to now but it seems it is taking from the beginning of hour.

For ex: Suppose the search returns result at 10:36:18 AM and I get an alert at 12:36:18 PM . But I think it is searching from the beginning of hour that is 10 AM so I keep on getting continuous alert. Can You please suggest anything .

0 Karma

somesoni2
Revered Legend

When the search frequency and time range are different (search runs every minute but looks back 2 hrs every time) there are events which will be overlapping in multiple alert execution, hence duplicate alerts. Best option is to keep them in sync, as suggested by @dineshraj9. Also, unless it's really critical, setup a less frequent alert like every 10 or 15 mins instead of every minute. Also, in time range, allow some additional gap to account for indexing latency.

E.g. Search time range earliest=-12m@m latest=-2m@m with cron as 2/10 * * * *

loveforsplunk
Explorer

oh .. I get it now .. Thanks a lot for instant reply. And also thanks to Dinesh .. I did not get what he wanted to express 😉 Sorry Dinesh but Thanks to both of you. 🙂 I am very new to Splunk so asking silly questions . Hope you didn't mind 🙂

Get Updates on the Splunk Community!

Updated Team Landing Page in Splunk Observability

We’re making some changes to the team landing page in Splunk Observability, based on your feedback. The ...

New! Splunk Observability Search Enhancements for Splunk APM Services/Traces and ...

Regardless of where you are in Splunk Observability, you can search for relevant APM targets including service ...

Webinar Recap | Revolutionizing IT Operations: The Transformative Power of AI and ML ...

The Transformative Power of AI and ML in Enhancing Observability   In the realm of IT operations, the ...