All Apps and Add-ons

Alert Manager Not Creating Incidents

tkw03
Communicator

Hello

 

After upgrading from and earlier version to 3.0.9, since i saw there were people having the JavaScript issue I was trying to fix, the app isnt creating incidents anymore.

I found this in the alert_manager_scheduler.log which is the only log of alert manager that has logs. I have checked the kvstore, its ready on all shc members but none of the alert metadata is getting created.

 

 

2022-08-17 13:42:19,996 WARNING pid="5761" logger="alert_manager_scheduler" message="KV Store is not yet available, sleeping for 1s." (alert_manager_scheduler.py:62)

 

 

 

The alerts run, they try to send, but get this in the splunkd.log

 

 

08-17-2022 13:46:05.489 -0400 INFO  sendmodalert [25767 AlertNotifierWorker-0] - Invoking modular alert action=alert_manager for search="Widows logging" sid="scheduler__<user>__search__RMD5467d08babc5954da_at_1660758360_111_64D51C26-A29A-41E8-917F-9211B53D56B5" in app="search" owner="<user>" type="saved"
08-17-2022 13:46:06.095 -0400 ERROR sendmodalert [25767 AlertNotifierWorker-0] - action=alert_manager STDERR -  Traceback (most recent call last):
08-17-2022 13:46:06.095 -0400 ERROR sendmodalert [25767 AlertNotifierWorker-0] - action=alert_manager STDERR -    File "/opt/splunk/etc/apps/alert_manager/bin/alert_manager.py", line 574, in <module>
08-17-2022 13:46:06.095 -0400 ERROR sendmodalert [25767 AlertNotifierWorker-0] - action=alert_manager STDERR -      config = getIncidentSettings(payload, settings, search_name, sessionKey)
08-17-2022 13:46:06.095 -0400 ERROR sendmodalert [25767 AlertNotifierWorker-0] - action=alert_manager STDERR -    File "/opt/splunk/etc/apps/alert_manager/bin/alert_manager.py", line 484, in getIncidentSettings
08-17-2022 13:46:06.095 -0400 ERROR sendmodalert [25767 AlertNotifierWorker-0] - action=alert_manager STDERR -      if ('impact' in result or result['impact'] != ''):
08-17-2022 13:46:06.095 -0400 ERROR sendmodalert [25767 AlertNotifierWorker-0] - action=alert_manager STDERR -  KeyError: 'impact'
08-17-2022 13:46:06.142 -0400 INFO  sendmodalert [25767 AlertNotifierWorker-0] - action=alert_manager - Alert action script completed in duration=651 ms with exit code=1
08-17-2022 13:46:06.142 -0400 WARN  sendmodalert [25767 AlertNotifierWorker-0] - action=alert_manager - Alert action script returned error code=1
08-17-2022 13:46:06.142 -0400 ERROR SearchScheduler [25767 AlertNotifierWorker-0] - Error in 'sendalert' command: Alert script returned error code 1., search='sendalert alert_manager results_file="/opt/splunk/var/run/splunk/dispatch/scheduler__<user>__search__RMD5467d08babc5954da_at_1660758360_111_64D51C26-A29A-41E8-917F-9211B53D56B5/results.csv.gz" results_link="https://<host>:8000/app/search/@go?sid=scheduler__<user>__search__RMD5467d08babc5954da_at_1660758360_111_64D51C26-A29A-41E8-917F-9211B53D56B5"'

 

 

 

does anyone have any idea what might be going on?


Thanks for your assistance

Labels (3)
0 Karma
Get Updates on the Splunk Community!

Splunk Education - Fast Start Program!

Welcome to Splunk Education! Splunk training programs are designed to enable you to get started quickly and ...

Five Subtly Different Ways of Adding Manual Instrumentation in Java

You can find the code of this example on GitHub here. Please feel free to star the repository to keep in ...

New Splunk APM Enhancements Help Troubleshoot Your MySQL and NoSQL Databases Faster

Splunk Observability has two new enhancements to make it quicker and easier to troubleshoot slow or frequently ...