We restored a backup. Splunk is back at version 9.2.2 and everything is working like before. I've checked the Alert Manager before upgrading and is should be compatible with 9.3.0. We will give it another try in a few weeks. Once again, thanks for your help. Much appreciated. ... View more

We will continue the restore for now, as our security team is pushing me. I will check your information when I try another upgrade. I really appreciate your help! Thank you! ... View more

Same problem. But on the other server it has the same permissions and it works without any problems. -rwxr--r-- 1 root root  5743 Jul 30 14:26 alertqueue_consumer.py -rwxr--r-- 1 root root 10515 Jul 30 14:26 command_ameenrich.py -rwxr--r-- 1 root root  4633 Jul 30 14:26 command_ameevents.py -rwxr--r-- 1 root root 11458 Jul 30 14:26 create_alert.py -rwxr--r-- 1 root root  1207 Jul 30 14:26 _env.py -rwxr--r-- 1 root root  6027 Jul 30 14:26 handler_license.py -rwxr--r-- 1 root root  4405 Jul 30 14:26 handler_logging.py -rwxr--r-- 1 root root  3192 Jul 30 14:26 handler_minit.py -rwxr--r-- 1 root root  3384 Jul 30 14:26 handler_proxy.py -rwxr--r-- 1 root root   441 Jul 30 14:26 handler.py -rwxr--r-- 1 root root  3578 Jul 30 14:26 handler_role_utils.py -rwxr--r-- 1 root root  5497 Jul 30 14:26 house_keeping.py -rwxr--r-- 1 root root   273 Jul 30 14:26 __init__.py -rwxr--r-- 1 root root  3832 Jul 30 14:26 notificationqueue_consumer.py drwxr-xr-x 2 root root  4096 Jul 30 14:26 persistconn drwx--x--- 2 root root  4096 Jul 30 14:26 __pycache__ -rwxr--r-- 1 root root  3615 Jul 30 14:26 tag_keeping.py We will restore a backup as we want the system to be up again. I will try an upgrade in a few weeks. If anything changes, I will update the post. Thank you for your help! ... View more

We have 2 servers, they look the same at the "not upgrade" server. -rw-r--r-- 1 root root 5743 Jul 30 14:26 alertqueue_consumer.py -rw-r--r-- 1 root root 10515 Jul 30 14:26 command_ameenrich.py -rw-r--r-- 1 root root 4633 Jul 30 14:26 command_ameevents.py -rw-r--r-- 1 root root 11458 Jul 30 14:26 create_alert.py -rw-r--r-- 1 root root 1207 Jul 30 14:26 _env.py -rw-r--r-- 1 root root 6027 Jul 30 14:26 handler_license.py -rw-r--r-- 1 root root 4405 Jul 30 14:26 handler_logging.py -rw-r--r-- 1 root root 3192 Jul 30 14:26 handler_minit.py -rw-r--r-- 1 root root 3384 Jul 30 14:26 handler_proxy.py -rw-r--r-- 1 root root 441 Jul 30 14:26 handler.py -rw-r--r-- 1 root root 3578 Jul 30 14:26 handler_role_utils.py -rw-r--r-- 1 root root 5497 Jul 30 14:26 house_keeping.py -rw-r--r-- 1 root root 273 Jul 30 14:26 __init__.py -rw-r--r-- 1 root root 3832 Jul 30 14:26 notificationqueue_consumer.py drwxr-xr-x 2 root root 4096 Jul 30 14:26 persistconn drwx--x--- 2 root root 4096 Jul 30 14:26 __pycache__ -rw-r--r-- 1 root root 3615 Jul 30 14:26 tag_keeping.py ... View more

root@dc-splunk01://opt/splunk/etc/apps/alert_manager_enterprise/bin# /opt/splunk/bin/splunk cmd python3 house_keeping.py --scheme Traceback (most recent call last): File "/opt/splunk/etc/apps/alert_manager_enterprise/bin/house_keeping.py", line 21, in <module> from datapunctum.factory_logger import Logger File "/opt/splunk/etc/apps/alert_manager_enterprise/bin/../lib/datapunctum/factory_logger.py", line 10, in <module> from pydantic import BaseModel File "/opt/splunk/etc/apps/alert_manager_enterprise/bin/../lib/pydantic/__init__.py", line 372, in __getattr__ module = import_module(module_name, package=package) File "/opt/splunk/lib/python3.9/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "/opt/splunk/etc/apps/alert_manager_enterprise/bin/../lib/pydantic/main.py", line 11, in <module> import pydantic_core File "/opt/splunk/etc/apps/alert_manager_enterprise/bin/../lib/pydantic_core/__init__.py", line 6, in <module> from ._pydantic_core import ( ModuleNotFoundError: No module named 'pydantic_core._pydantic_core' New install of the app gave the same result. ... View more