Alerting

Real time alerts

brettcave
Builder

I originally posted this because our alerts weren't working, and I wanted to confirm the syntax for multiple recipients. It seems that our alerts still aren't working (not getting email notification or showing in the alert manager). One of the comments posted in the other question was that alltime realtime (rt / rt) alerts should not be configured, and we had a number of them. So what is the best way to configure real-time searches then? Our use-case is that we want to be notified as soon as certain events occur.

I went in to all the "rt rt" searches, and changed them to "rt-1m / rt-0m" time frames, with condition "always" and alert mode "per-result" with some relevant field throttling, but after running some tests, we're not getting the notifications as expected.

I'm considering combining all of our rt/rt searches into 1 monster query (we had about 15 odd searches) with the use of ()'s and ANDs / ORs, so that one search matches all (although identifying which condition triggered it by subject will be a nightmare, unless we have some crazy eval + case to inject a label).

What is the best approach for configuring searches to notify email addresses as certain events occur?

Tags (2)
0 Karma

yannK
Splunk Employee
Splunk Employee

Remember, if you use realtime rt-1m to rt
or scheduled every minute with -1m@m to now
Then all events coming with : 1minute delay, or from the future (yes it happens a lot with clock differences) will no be included in the search,

Preventively, you should estimate your average delay to figure the best time windows.
* | eval delay_sec=_indextime-_time | stats min(delay_sec) avg(delay_sec) max(delay_sec) by host source

by example, if the delay goes up to 3 minutes, use rt-3, rt, or a search running every minute over 3 minutes ago -4m@m to -3m@m

brettcave
Builder

the SMTP server we use requires authentication. Doesn't look like the sendemail command supports authentication.

0 Karma

linu1988
Champion

I found something. Could you try this search instead of the one you are using. The earliest one is not sending any mail as it doesn't have a condition to match the result.

sourcetype="source"|table _time,host,RequestURI|where RequestURI="/ping"

Configure this in the alert it will work for sure. Other configurations are correct.

linu1988
Champion

It doesn't really depend on the indexer restart or you don't have to.It's some other reason, works well if the search query with a valid condition. Could you post your query?

0 Karma

brettcave
Builder

I have restarted the splunk indexer, and still don't get notifications from my alert.

0 Karma

brettcave
Builder

Are there any limitations in terms of firing an alert based on which index is being used? My splunk 5.0.4 is now sending alerts for the test case I configured, but not for my actual alert. And they are both configured EXACTLY the same, except for the actual fields they use for & the index. test case = "main" and real = "my_custom_index"

rt-1h / rt-0, alert "always", "once per result", throttled for 1 hour based on JobExecId. 

If I click the search from the Searches & reports drop down, I see the results. But still no alert in Alert Manager and no email.

0 Karma

brettcave
Builder

and the alerts come flying in. Thanks again jtacy.

0 Karma

brettcave
Builder

(we're running 5.0.2)

0 Karma

brettcave
Builder

thanks jtacy - that must be it. I am scheduling an upgrade and will post back.

0 Karma

jtacy
Builder

Hmm...could you be hitting this issue that was fixed in 5.0.3?
Real Time Alerts not working consistently in 5.0.2. (SPL-62129)

If this is consistently reproducible on 5.0.4, it seems like something that Support might want to take on. The idea that real time alerts would break without warning is discomforting, particularly considering the types of events those alerts are likely to be used for.

brettcave
Builder

I edited the alert to my original configuration. Triggered the condition. Didn't receive a notification. Restarted splunk. Triggered condition again. The notification comes through. It looks like editing a real-time search with an alert breaks the alert.

0 Karma

brettcave
Builder

Ok - the reason the alert wasn't firing relates to an issue we've found with Splunk before. Sometimes, if you edit an alert, all notifications stop. I have now restarted Splunk server, and the alert fires.

0 Karma

brettcave
Builder

Nope. Still not. Search is sourcetype="mysourcetype" | table _time RequestURI | where RequestURI="/ping". If I run the search from the drop-down, I see the result. No alert is fired (i.e. no email or no event in the alert manager)

0 Karma

linu1988
Champion

Could you run this in your search replacing the correct values?

sourcetype="source" RequestURI="/ping" |table _time,host,RequestURI| sendemail to=abc@abc.com server=smtp_server sendresults=true format=html inline=true

Choose a timeperiod where you have result.

Let us know if you get the email for the result.

0 Karma

brettcave
Builder

I've added screenshot of the config as well as seeing a result when I'm running the search. nothing in alert manager. no email response. other emails on the system are working (e.g. scheduled pdf report view). My email address is pretty standard - brett at mycompany dot com.

0 Karma

brettcave
Builder

config

alt text

0 Karma

brettcave
Builder

it's configured in search. RequestURI is an extracted field. It has full view permissions. http://answers.splunk.com/answers/99570/whats-the-correct-format-for-multiple-email-addresses-in-an-... - answer stats "comma or semi-colon to seperate email addresses" - I have changed the alert to use 1 email address. still not registering.

0 Karma

linu1988
Champion

In which app the alert is configured?

Is the Request_URI is an extracted field?

0 Karma

linu1988
Champion

Emails should be separated by ";"
action.email = 1
action.email.cc = abc.abc@com;abc.abc@com
action.email.from = abc.abc@com
action.email.inline = 1
action.email.sendresults = 1
action.email.to = abc.abc@com
alert.digest_mode = False
alert.expires = 30m
alert.suppress = 1
alert.suppress.fields = host
alert.suppress.period = 1h
alert.track = 0
cron_schedule = * * * * *
dispatch.earliest_time = rt
dispatch.latest_time = rt
displayview = flashtimeline
enableSched = 1
quantity = 0
relation = greater than
request.ui_dispatch_view = flashtimeline
search = .....

0 Karma

brettcave
Builder

Created a test case. search is RequestURI="/ping" | table _time RequestURI. Created 2 alerts: 1 alltime/real-time with no throttling and another realtime/1minute rolling window with "number of events" > 0, with alert mode "once per search" and 60 second throttling. Both alerts have tracking enabled with 24 hour expiration.

I hit the URI to trigger the event - GET /ping. I am running both searches in 2 splunk windows. Both manually running searches show the hit. I don't get a notification. The alert manager doesn't show anything. Both alerts have 2 email addresses configured (comma sep.)

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...