I originally posted this because our alerts weren't working, and I wanted to confirm the syntax for multiple recipients. It seems that our alerts still aren't working (not getting email notification or showing in the alert manager). One of the comments posted in the other question was that alltime realtime (rt / rt) alerts should not be configured, and we had a number of them. So what is the best way to configure real-time searches then? Our use-case is that we want to be notified as soon as certain events occur.
I went in to all the "rt rt" searches, and changed them to "rt-1m / rt-0m" time frames, with condition "always" and alert mode "per-result" with some relevant field throttling, but after running some tests, we're not getting the notifications as expected.
I'm considering combining all of our rt/rt searches into 1 monster query (we had about 15 odd searches) with the use of ()'s and ANDs / ORs, so that one search matches all (although identifying which condition triggered it by subject will be a nightmare, unless we have some crazy eval + case to inject a label).
What is the best approach for configuring searches to notify email addresses as certain events occur?
Remember, if you use realtime rt-1m to rt
or scheduled every minute with -1m@m to now
Then all events coming with : 1minute delay, or from the future (yes it happens a lot with clock differences) will no be included in the search,
Preventively, you should estimate your average delay to figure the best time windows.
* | eval delay_sec=_indextime-_time | stats min(delay_sec) avg(delay_sec) max(delay_sec) by host source
by example, if the delay goes up to 3 minutes, use rt-3, rt, or a search running every minute over 3 minutes ago -4m@m to -3m@m
I found something. Could you try this search instead of the one you are using. The earliest one is not sending any mail as it doesn't have a condition to match the result.
sourcetype="source"|table _time,host,RequestURI|where RequestURI="/ping"
Configure this in the alert it will work for sure. Other configurations are correct.
Are there any limitations in terms of firing an alert based on which index is being used? My splunk 5.0.4 is now sending alerts for the test case I configured, but not for my actual alert. And they are both configured EXACTLY the same, except for the actual fields they use for & the index. test case = "main" and real = "my_custom_index"
rt-1h / rt-0, alert "always", "once per result", throttled for 1 hour based on JobExecId.
If I click the search from the Searches & reports drop down, I see the results. But still no alert in Alert Manager and no email.
Hmm...could you be hitting this issue that was fixed in 5.0.3?
Real Time Alerts not working consistently in 5.0.2. (SPL-62129)
If this is consistently reproducible on 5.0.4, it seems like something that Support might want to take on. The idea that real time alerts would break without warning is discomforting, particularly considering the types of events those alerts are likely to be used for.
I edited the alert to my original configuration. Triggered the condition. Didn't receive a notification. Restarted splunk. Triggered condition again. The notification comes through. It looks like editing a real-time search with an alert breaks the alert.
Ok - the reason the alert wasn't firing relates to an issue we've found with Splunk before. Sometimes, if you edit an alert, all notifications stop. I have now restarted Splunk server, and the alert fires.
Nope. Still not. Search is
sourcetype="mysourcetype" | table _time RequestURI | where RequestURI="/ping". If I run the search from the drop-down, I see the result. No alert is fired (i.e. no email or no event in the alert manager)
Could you run this in your search replacing the correct values?
sourcetype="source" RequestURI="/ping" |table _time,host,RequestURI| sendemail email@example.com server=smtp_server sendresults=true format=html inline=true
Choose a timeperiod where you have result.
Let us know if you get the email for the result.
I've added screenshot of the config as well as seeing a result when I'm running the search. nothing in alert manager. no email response. other emails on the system are working (e.g. scheduled pdf report view). My email address is pretty standard - brett at mycompany dot com.
it's configured in search. RequestURI is an extracted field. It has full view permissions. http://answers.splunk.com/answers/99570/whats-the-correct-format-for-multiple-email-addresses-in-an-... - answer stats "comma or semi-colon to seperate email addresses" - I have changed the alert to use 1 email address. still not registering.
Emails should be separated by ";"
action.email = 1
action.email.cc = abc.abc@com;abc.abc@com
action.email.from = abc.abc@com
action.email.inline = 1
action.email.sendresults = 1
action.email.to = abc.abc@com
alert.digest_mode = False
alert.expires = 30m
alert.suppress = 1
alert.suppress.fields = host
alert.suppress.period = 1h
alert.track = 0
cron_schedule = * * * * *
dispatch.earliest_time = rt
dispatch.latest_time = rt
displayview = flashtimeline
enableSched = 1
quantity = 0
relation = greater than
request.ui_dispatch_view = flashtimeline
search = .....
Created a test case. search is
RequestURI="/ping" | table _time RequestURI. Created 2 alerts: 1 alltime/real-time with no throttling and another realtime/1minute rolling window with "number of events" > 0, with alert mode "once per search" and 60 second throttling. Both alerts have tracking enabled with 24 hour expiration.
I hit the URI to trigger the event - GET /ping. I am running both searches in 2 splunk windows. Both manually running searches show the hit. I don't get a notification. The alert manager doesn't show anything. Both alerts have 2 email addresses configured (comma sep.)