Alerting

How to troubleshoot real-time alerts not working?

Splunk Employee
Splunk Employee

Hello,
I am having a hard time trying to pin down why most of my real-time alerts have stopped working. I have looked into scheduler.log and python.log, and did not find much insightful details to the problem. Here are the symptoms:

  1. Only real-time alerts are not appearing to fire
  2. Non-realtime alerts appear to be fine as I am still getting alert emails
  3. Once splunk is restarted, some rt alerts appear to be firing; then eventually stopped
1 Solution

Splunk Employee
Splunk Employee

Please follow the steps below to debug this issue:

1.) Backup and edit $SPLUNK_HOME/etc/log.cfg and locate the following entry:

category.SavedSplunker=INFO,scheduler

2.) change the INFO key to DEBUG (ie. category.SavedSplunker=DEBUG,scheduler)
3.) save the changes and restart splunk

4.) try to generate an event that should trigger an alert, but not happening.

5.) the debug messages (with diagnostics) should be written into $SPLUNK_HOME/var/log/splunk/scheduler.log

6) Identify the root cause, fix it and revert back the changes made in step 2 and restart Splunk to get rid of the DEBUG messages that you hopefully don't need any more

Hope This Helps!

View solution in original post

Path Finder

Same situation, what was the solution for you here?

0 Karma

Splunk Employee
Splunk Employee

Please follow the steps below to debug this issue:

1.) Backup and edit $SPLUNK_HOME/etc/log.cfg and locate the following entry:

category.SavedSplunker=INFO,scheduler

2.) change the INFO key to DEBUG (ie. category.SavedSplunker=DEBUG,scheduler)
3.) save the changes and restart splunk

4.) try to generate an event that should trigger an alert, but not happening.

5.) the debug messages (with diagnostics) should be written into $SPLUNK_HOME/var/log/splunk/scheduler.log

6) Identify the root cause, fix it and revert back the changes made in step 2 and restart Splunk to get rid of the DEBUG messages that you hopefully don't need any more

Hope This Helps!

View solution in original post

State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!