topic Re: Real time alerts in Alerting

Real time alerts

brettcave — Tue, 20 Aug 2013 15:39:17 GMT

I originally posted this because our alerts weren't working, and I wanted to confirm the syntax for multiple recipients. It seems that our alerts still aren't working (not getting email notification or showing in the alert manager). One of the comments posted in the other question was that alltime realtime (rt / rt) alerts should not be configured, and we had a number of them. So what is the best way to configure real-time searches then? Our use-case is that we want to be notified as soon as certain events occur.

I went in to all the "rt rt" searches, and changed them to "rt-1m / rt-0m" time frames, with condition "always" and alert mode "per-result" with some relevant field throttling, but after running some tests, we're not getting the notifications as expected.

I'm considering combining all of our rt/rt searches into 1 monster query (we had about 15 odd searches) with the use of ()'s and ANDs / ORs, so that one search matches all (although identifying which condition triggered it by subject will be a nightmare, unless we have some crazy eval + case to inject a label).

What is the best approach for configuring searches to notify email addresses as certain events occur?

Re: Real time alerts

lukejadamec — Tue, 20 Aug 2013 23:27:48 GMT

When you look at the report//alert in Manager, what does the scheduled time say?

Re: Real time alerts

grijhwani — Wed, 21 Aug 2013 03:18:00 GMT

Splunk is not the ideal tool for literal "real-time" alerting. If you need truly real-time alerting you need a real-time monitoring platform (Nagios or similar under Linux, for instance). That said unless you are using something of the ilk of SNMP traps to initiate alerts, nothing is ever truly real time, as you are inevitably relying on a regular polling of whatever conditional semaphores you are monitoring, even if that polling is something like once a second.

The best you can really achieve with Splunk is regular searches running at short intervals over short time spans (e.g. scheduled to run every minute, and only cover a span of a minute - or possibly two just to ensure overlap and that nothing falls through the cracks).

Really, it comes down to just how instantaneous you need your alert to be. After all, if you are relying on e-mail alerts you could conceivably fall foul of delivery delays.

Re: Real time alerts

brettcave — Wed, 21 Aug 2013 08:27:11 GMT

We originally used nagios / zabbix as our monitoring system. Those tools are great for OS / platform monitoring (although the *nix app works pretty well in splunk too). We've tried to consolidate our logging in splunk (instead of managing more than 1 app) - so for now, we are looking to get close to real time monitoring. By "close", i mean notified within a minute or 2 (immediate not necessary).

so with that in mind, use a -2m / 0 range scheduled to run every minute, with a 1 minute suppression based on a unique field (e.g. a run id for a job) would be a good approach?

Re: Real time alerts

brettcave — Wed, 21 Aug 2013 08:27:29 GMT

Schedule this alert: checked
Alert Condition: always

Re: Real time alerts

linu1988 — Wed, 21 Aug 2013 09:10:22 GMT

I would like to know the search and the throttling parameters. The real time alerts work fine, i had struggled with it but i got it worked with precision. So do explain with the search and condition so that we can look at. Probably you can show us in the image.

Re: Real time alerts

brettcave — Wed, 21 Aug 2013 09:22:59 GMT

out of the 15 searches, it depends on the search. For example, we have one that has:

For example, one alert that we want to be notified when a user of our application triggers a certain condition has "Once per result" with throttling of 1 hour based on UserID.

However, we have another alert that monitors logs from the application to the database. We don't want to throttle this event though, every time the application has an error connecting to the database, we want it to email us. We currently have rt-1h to rt-0 with condition of "number of events" > 0 and 1 hour throttling based on "host"

Re: Real time alerts

linu1988 — Wed, 21 Aug 2013 09:45:52 GMT

It doesn't depend on whether you have 15/20 realtime searches, it's about how it's configured.

Are you getting any mail for any of the configured alert?

If not these are the possible cause may happen:
The sendmail.py file which sends the mail may be corrupt.
The alert condition doesn't match any event.
The throttling parameter is not the actual field name
The SMTP server is not configured correctly.

simple way to test from search app:

...| sendemail to=abc@abc.com server=smtp_server sendresults=true format=html inline=true

test it under http://server:8000/en-US/app/App/flashtimeline

Re: Real time alerts

brettcave — Wed, 21 Aug 2013 09:49:30 GMT

I don't think it's at the SMTP level, because i have tracking enabled, and the alltime / realtime (rt rt) searches weren't even showing in the alert manager.

The alert condition SHOULD match an event - if I open the search from the "Searches and Reports" drop down, then I can see the events showing. However, its something to do with rt/rt config that seems to be breaking it. I've been fiddling around, but am busy configuring a specific test case to check what happens.

Re: Real time alerts

brettcave — Wed, 21 Aug 2013 10:01:40 GMT

Created a test case. search is RequestURI="/ping" | table _time RequestURI. Created 2 alerts: 1 alltime/real-time with no throttling and another realtime/1minute rolling window with "number of events" > 0, with alert mode "once per search" and 60 second throttling. Both alerts have tracking enabled with 24 hour expiration.

I hit the URI to trigger the event - GET /ping. I am running both searches in 2 splunk windows. Both manually running searches show the hit. I don't get a notification. The alert manager doesn't show anything. Both alerts have 2 email addresses configured (comma sep.)

Re: Real time alerts

linu1988 — Mon, 28 Sep 2020 14:36:57 GMT

Emails should be separated by ";"
action.email = 1
action.email.cc = abc.abc@com;abc.abc@com
action.email.from = abc.abc@com
action.email.inline = 1
action.email.sendresults = 1
action.email.to = abc.abc@com
alert.digest_mode = False
alert.expires = 30m
alert.suppress = 1
alert.suppress.fields = host
alert.suppress.period = 1h
alert.track = 0
cron_schedule = * * * * *
dispatch.earliest_time = rt
dispatch.latest_time = rt
displayview = flashtimeline
enableSched = 1
quantity = 0
relation = greater than
request.ui_dispatch_view = flashtimeline
search = .....

Re: Real time alerts

linu1988 — Wed, 21 Aug 2013 10:15:49 GMT

In which app the alert is configured?

Is the Request_URI is an extracted field?

Re: Real time alerts

brettcave — Wed, 21 Aug 2013 10:26:26 GMT

it's configured in search. RequestURI is an extracted field. It has full view permissions. http://answers.splunk.com/answers/99570/whats-the-correct-format-for-multiple-email-addresses-in-an-alert - answer stats "comma or semi-colon to seperate email addresses" - I have changed the alert to use 1 email address. still not registering.

Re: Real time alerts

brettcave — Wed, 21 Aug 2013 10:38:12 GMT

Re: Real time alerts

brettcave — Wed, 21 Aug 2013 10:39:27 GMT

I've added screenshot of the config as well as seeing a result when I'm running the search. nothing in alert manager. no email response. other emails on the system are working (e.g. scheduled pdf report view). My email address is pretty standard - brett at mycompany dot com.

Re: Real time alerts

linu1988 — Mon, 28 Sep 2020 14:37:00 GMT

Could you run this in your search replacing the correct values?

sourcetype="source" RequestURI="/ping" |table _time,host,RequestURI| sendemail to=abc@abc.com server=smtp_server sendresults=true format=html inline=true

Choose a timeperiod where you have result.

Let us know if you get the email for the result.

Re: Real time alerts

linu1988 — Wed, 21 Aug 2013 11:22:01 GMT

I found something. Could you try this search instead of the one you are using. The earliest one is not sending any mail as it doesn't have a condition to match the result.

sourcetype="source"|table _time,host,RequestURI|where RequestURI="/ping"

Configure this in the alert it will work for sure. Other configurations are correct.

Re: Real time alerts

grijhwani — Wed, 21 Aug 2013 11:47:51 GMT

Here's a thought: you could consider integrate Splunk with Nagios passive checks and rely on that engine to handle the actual alerting. I have not done it myself, but I know it has been done.

Re: Real time alerts

brettcave — Wed, 21 Aug 2013 12:32:07 GMT

the SMTP server we use requires authentication. Doesn't look like the sendemail command supports authentication.

Re: Real time alerts

brettcave — Wed, 21 Aug 2013 12:40:55 GMT

Nope. Still not. Search is sourcetype="mysourcetype" | table _time RequestURI | where RequestURI="/ping". If I run the search from the drop-down, I see the result. No alert is fired (i.e. no email or no event in the alert manager)