Alerting

How to troubleshoot real-time alerts not working?

splunkIT
Splunk Employee
Splunk Employee

Hello,
I am having a hard time trying to pin down why most of my real-time alerts have stopped working. I have looked into scheduler.log and python.log, and did not find much insightful details to the problem. Here are the symptoms:

  1. Only real-time alerts are not appearing to fire
  2. Non-realtime alerts appear to be fine as I am still getting alert emails
  3. Once splunk is restarted, some rt alerts appear to be firing; then eventually stopped
1 Solution

uuppuluri_splun
Splunk Employee
Splunk Employee

Please follow the steps below to debug this issue:

1.) Backup and edit $SPLUNK_HOME/etc/log.cfg and locate the following entry:

category.SavedSplunker=INFO,scheduler

2.) change the INFO key to DEBUG (ie. category.SavedSplunker=DEBUG,scheduler)
3.) save the changes and restart splunk

4.) try to generate an event that should trigger an alert, but not happening.

5.) the debug messages (with diagnostics) should be written into $SPLUNK_HOME/var/log/splunk/scheduler.log

6) Identify the root cause, fix it and revert back the changes made in step 2 and restart Splunk to get rid of the DEBUG messages that you hopefully don't need any more

Hope This Helps!

View solution in original post

salem34
Path Finder

Same situation, what was the solution for you here?

0 Karma

uuppuluri_splun
Splunk Employee
Splunk Employee

Please follow the steps below to debug this issue:

1.) Backup and edit $SPLUNK_HOME/etc/log.cfg and locate the following entry:

category.SavedSplunker=INFO,scheduler

2.) change the INFO key to DEBUG (ie. category.SavedSplunker=DEBUG,scheduler)
3.) save the changes and restart splunk

4.) try to generate an event that should trigger an alert, but not happening.

5.) the debug messages (with diagnostics) should be written into $SPLUNK_HOME/var/log/splunk/scheduler.log

6) Identify the root cause, fix it and revert back the changes made in step 2 and restart Splunk to get rid of the DEBUG messages that you hopefully don't need any more

Hope This Helps!

Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...