Alert Manager Not Creating Incidents

[tkw03]

Hello

After upgrading from and earlier version to 3.0.9, since i saw there were people having the JavaScript issue I was trying to fix, the app isnt creating incidents anymore.

I found this in the alert_manager_scheduler.log which is the only log of alert manager that has logs. I have checked the kvstore, its ready on all shc members but none of the alert metadata is getting created.

2022-08-17 13:42:19,996 WARNING pid="5761" logger="alert_manager_scheduler" message="KV Store is not yet available, sleeping for 1s." (alert_manager_scheduler.py:62)

The alerts run, they try to send, but get this in the splunkd.log

08-17-2022 13:46:05.489 -0400 INFO  sendmodalert [25767 AlertNotifierWorker-0] - Invoking modular alert action=alert_manager for search="Widows logging" sid="scheduler__<user>__search__RMD5467d08babc5954da_at_1660758360_111_64D51C26-A29A-41E8-917F-9211B53D56B5" in app="search" owner="<user>" type="saved"
08-17-2022 13:46:06.095 -0400 ERROR sendmodalert [25767 AlertNotifierWorker-0] - action=alert_manager STDERR -  Traceback (most recent call last):
08-17-2022 13:46:06.095 -0400 ERROR sendmodalert [25767 AlertNotifierWorker-0] - action=alert_manager STDERR -    File "/opt/splunk/etc/apps/alert_manager/bin/alert_manager.py", line 574, in <module>
08-17-2022 13:46:06.095 -0400 ERROR sendmodalert [25767 AlertNotifierWorker-0] - action=alert_manager STDERR -      config = getIncidentSettings(payload, settings, search_name, sessionKey)
08-17-2022 13:46:06.095 -0400 ERROR sendmodalert [25767 AlertNotifierWorker-0] - action=alert_manager STDERR -    File "/opt/splunk/etc/apps/alert_manager/bin/alert_manager.py", line 484, in getIncidentSettings
08-17-2022 13:46:06.095 -0400 ERROR sendmodalert [25767 AlertNotifierWorker-0] - action=alert_manager STDERR -      if ('impact' in result or result['impact'] != ''):
08-17-2022 13:46:06.095 -0400 ERROR sendmodalert [25767 AlertNotifierWorker-0] - action=alert_manager STDERR -  KeyError: 'impact'
08-17-2022 13:46:06.142 -0400 INFO  sendmodalert [25767 AlertNotifierWorker-0] - action=alert_manager - Alert action script completed in duration=651 ms with exit code=1
08-17-2022 13:46:06.142 -0400 WARN  sendmodalert [25767 AlertNotifierWorker-0] - action=alert_manager - Alert action script returned error code=1
08-17-2022 13:46:06.142 -0400 ERROR SearchScheduler [25767 AlertNotifierWorker-0] - Error in 'sendalert' command: Alert script returned error code 1., search='sendalert alert_manager results_file="/opt/splunk/var/run/splunk/dispatch/scheduler__<user>__search__RMD5467d08babc5954da_at_1660758360_111_64D51C26-A29A-41E8-917F-9211B53D56B5/results.csv.gz" results_link="https://<host>:8000/app/search/@go?sid=scheduler__<user>__search__RMD5467d08babc5954da_at_1660758360_111_64D51C26-A29A-41E8-917F-9211B53D56B5"'

does anyone have any idea what might be going on?

Thanks for your assistance