topic Re: Real Time Alerting in Alerting

Real Time Alerting

tnconners — Fri, 11 Oct 2013 18:22:54 GMT

I'm working on configuring some basic alerts for the a system. This is splunk 5.0.2 on Windows 2008 R2.

The search is very simple:

Source = "E:\Program Files*" High

which returns results every time, now before fine tuning the search I wan to confirm that the alerts will fire correctly through alert manager and SMTP.

My parameters for the alert is as follows:
Start Time rt End time rt

Alert
Condition Always

Alert Mode once per result

no throttling

Expiration 24 hours

Severity High

Send email "valid email address with subject etc"

Tracking enabled

This alert should be overloading my inbox with emails, but it's not showing in alert manager even. The only thing I can think of is we currently have license violations on this instance, but searching and alerting are not yet disabled. The capacity for the day is blown though.

Any help is appreciated!

EDIT: Turned out that we had way to many saved searches (that were no longer relevant since we are making out alerts generic) I cleared them out of the saved searches .conf file and things started running better. I also had upgraded from 5.02 to 5.05.

Thanks for your help everyone!

Re: Real Time Alerting

exd42062 — Sun, 13 Oct 2013 01:31:30 GMT

Splunk regulates your license usage by tracking license violations. If you go over 500 MB/day more than 3 times in a 30 day period, Splunk continues to index your data, but disables search functionality until you are back down to 3 or fewer warnings in the 30 day period.

Re: Real Time Alerting

lukejadamec — Sun, 13 Oct 2013 10:55:58 GMT

If the search works manually, then it is not a license issue. When you have to many violations in a 30 day period then you can't search at all.

Your start time should be rt-1m
Consider setting the alert condition to trigger on number of results greater than 1.

Don't test the search from the search app, test it by selecting Run from the Manager > Searches and Reports.

You can also reconfigure to run as a scheduled search that runs every minute, and trigger on number of results greater than 1.

Re: Real Time Alerting

jtacy — Sun, 13 Oct 2013 20:26:30 GMT

There was a problem with 5.0.2 that affected real time alerts and was fixed in 5.0.3. It's in the 5.0.3 release notes as "Real Time Alerts not working consistently in 5.0.2. (SPL-62129)". Might be worth taking a brief outage to upgrade to 5.0.5. Good luck!

Re: Real Time Alerting

tnconners — Mon, 14 Oct 2013 17:54:27 GMT

Tried all of your suggestions, Still no luck. I also upgraded as jtacy suggested. It seems like my scheduled searches are never starting. (I've watched the jobs screen).

Re: Real Time Alerting

lukejadamec — Tue, 15 Oct 2013 16:37:32 GMT

Try creating a new scheduled search from scratch. I had one that behaved like this once, and I had to create a new search to fix it.