Alert Manager Enterprise hang on

[Andras]

Hello Everyone!

I installed Splunk and Alert Manager Enterprise in Virtualbox for learning purposes (4cpu /8gb ram).

I configured AME via the documentation.

Health Check is green. I can send test alerts, they appear in the ame_default index.

However the alerts don't appear in the Events. Hang up forever.

I have some broken pipe errors, but they also appear in an another working environment.

Thank you for your help.

A

[seiimonn]

Hi Andras

• Which OS, Splunk version and AME version are you using?
• Is there anything visible in the browser developer console?
• Can you check if something loads if you open this URL in your browser? https://your-splunk:8000/en-GB/splunkd/__raw/services/ame_sysinfo

Are there any errors visible with 

index=_internal source=*ame* ERROR | table _time host source _raw

Please open a support case if you cannot share this information publicly.

Regards,
Simon

[Andras]

Hi @seiimonn !

I noticed that, every time i start AME Events, i get the following error.

I appreciate your help.

A.

2/20/25 8:40:34.605 AM

127.0.0.1 - splunk-system-user [20/Feb/2025:08:40:34.605 +0100] "GET /servicesNS/nobody/alert_manager_enterprise/messages/ame-index-resilience-default-error HTTP/1.0" 404 177 "-" "splunk-sdk-python/1.7.3" - - - 0ms

[Andras]

Hi @seiimonn !

Debian GNU/Linux 12 (bookworm)

Splunk Enterprise 9.0.0

AME 3.0.8.

Sysinfo:

{"uuid":"95c6740c-9e0b-42b1-b2b9-b78067db6677","status":200,"messages":[],"payload":{"tenant_list":[{"tenant_uid":"default","role":"admin"}],"is_admin":true,"is_app_admin":true,"products":[],"necessary_tasks":[],"legacy_installed":false,"environment":"on_premises","timezone":"UTC"}}

There ara no errors now, if i run this script:

index=_internal source=*ame* ERROR | table _time host source _raw

But maybe these ara interesting in the splunkd.log:

19/02/2025 19:34:15.506	02-19-2025 19:34:15.506 +0100 WARN HttpListener [1069 HttpDedicatedIoThread-4] - Socket error from 127.0.0.1:37790 while accessing /servicesNS/nobody/alert_manager_enterprise/properties/server: Broken pipe host = splunk source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd
	19/02/2025 19:34:08.828	2025-02-19 19:34:08,828 INFO [assist::supervisor_modular_input.py] [context] [build_supervisor_secrets] [22691] Secret load failed, key=tenant_id, error=[HTTP 404] https://127.0.0.1:8089/servicesNS/nobody/splunk_assist/storage/passwords/tenant_id?output_mode=json host = splunk source = /opt/splunk/var/log/splunk/splunk_assist_supervisor_modular_input.log sourcetype = splunk_assist_internal_log
	19/02/2025 19:34:06.362	02-19-2025 19:34:06.362 +0100 WARN HttpListener [1068 HttpDedicatedIoThread-3] - Socket error from 127.0.0.1:52422 while accessing /servicesNS/nobody/alert_manager_enterprise/properties/server: Broken pipe host = splunk source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd
	19/02/2025 19:33:56.500	2025-02-19 19:33:56.500 +0100 Trace-Id= type=METER, name=ch.qos.logback.core.Appender.error, count=3, m1_rate=3.527460396057507E-12, m5_rate=9.325633072421824E-5, m15_rate=7.016228689718483E-4, mean_rate=0.0019981503731937404, rate_unit=events/second host = splunk source = /opt/splunk/var/log/splunk/splunk_app_db_connect_health_metrics.log sourcetype = dbx_health_metrics
	19/02/2025 19:33:54.415	2025-02-19 19:33:54,415 INFO [assist::supervisor_modular_input.py] [context] [build_supervisor_secrets] [22467] Secret load failed, key=tenant_id, error=[HTTP 404] https://127.0.0.1:8089/servicesNS/nobody/splunk_assist/storage/passwords/tenant_id?output_mode=json host = splunk source = /opt/splunk/var/log/splunk/splunk_assist_supervisor_modular_input.log sourcetype = splunk_assist_internal_log

I use this script, to create test alerts:

| makeresults | eval user="World", src="192.168.0.1", action="create test event" | sendalert create_alert param.title="Hello $result.user$" param.template=default param.tenant_uid=default

I think there is nothing interesting on the browsers developer console. What do you think about that?

Thanks for your helping.

[gcusello]

Hi @Andras ,

you can see in Alert Manager App only alerts share at Global level, so you have to change the permissions in your alerts from App level to Global level.

Ciao.

Giuseppe

[Andras]

Hi @gcusello !

Also interesting that the alerts in the index seems good:

But the loading of the events in the Events dashboard never ending.

[gcusello]

Hi @Andras ,

if you haven't the issue about the Global Sharing of the Alerts, check the macros used in the dashboards, probably you have to specify the index where Notables are located.

You can do it opening the dashboard.

Ciao.

Giuseppe

[Andras]

Hi Giuseppe!

Thank you for your answer.

I double checked, but my alerts are alredy global.

I think there is another problem.

Thanks,

A