Alerting

Why am I still receiving alerts from Splunk even after disabling them?

Path Finder

Hi,

i am using Splunk 6.4.3. i have configured real-time alerts to verify. once it is done, i have disabled and deleted savedsearches.conf from the search head but still am getting alerts from Splunk. can anyone help on how to get rid of alerts?

0 Karma

New Member

is there any way to disable the real time alert option permanently??

0 Karma

Esteemed Legend

How are you determining that alerts are still firing? Let's say that you have the alert send you an email. Email is VERY laggy and it is possible that there are thousands of oustanding emails that, had the volume been slower/smaller, should have already been delivered, but because you were flooding emails, systems inbetween in the MTA chain are throttling delivery. You may see email for weeks to come that were sent BEFORE you turned off the alert. You can go to your Search Head and do this:

find $SPLUNK_HOME -name "savedsearches.conf"

Then look inside every file that you find and make SURE that anything like your search has disabled=1.

0 Karma

SplunkTrust
SplunkTrust

The real-time saved searches never stop running, so did you kill those searches after you deleted the savedsearches.conf entry? From the search head servers, you can do (for linux) psef splunk and see any process running for those real-time searches. If found, kill them.

0 Karma

Path Finder

i did ps -ef | grep splunk.

i could not find any process running with my savedsearches name.

0 Karma

SplunkTrust
SplunkTrust

It'll not be with your saved search name, there may be process which contains rt_ keyword and may have very old start time.

0 Karma

Path Finder

yeah i did check that but i could not find any process running.

0 Karma

SplunkTrust
SplunkTrust

Are you saying you modified the savedsearches.conf file from the command line? If so, did you also refresh or start Splunk? Most command line changes don't take effect until refresh/restart.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

Path Finder

yeah i did restart splunk. but still am getting lot of alerts

0 Karma

SplunkTrust
SplunkTrust

Did you modify the right file? Do the searches still appear in the GUI?

---
If this reply helps you, an upvote would be appreciated.
0 Karma

Path Finder

No There are no alerts or savaedsearches i can see in GUI

0 Karma