All Apps and Add-ons

Alert Manager Enterprise hang on

Andras
Loves-to-Learn

Hello Everyone!

I installed Splunk and Alert Manager Enterprise in Virtualbox for learning purposes (4cpu /8gb ram).

I configured AME via the documentation.

Health Check is green. I can send test alerts, they appear in the ame_default index.

Andras_1-1739969113061.pngAndras_2-1739969158389.png

 

However the alerts don't appear in the Events. Hang up forever.

Andras_0-1739969053285.png

I have some broken pipe errors, but they also appear in an another working environment.

Andras_3-1739969245319.png

Thank you for your help.

A

 

 

 

Labels (1)
Tags (1)
0 Karma

seiimonn
New Member

Hi Andras

Are there any errors visible with 

index=_internal source=*ame* ERROR | table _time host source _raw

 

Please open a support case if you cannot share this information publicly.

Regards,
Simon

 

0 Karma

Andras
Loves-to-Learn

Hi @seiimonn !

I noticed that, every time i start AME Events, i get the following error.

I appreciate your help.

A.

2/20/25
8:40:34.605 AM
 
127.0.0.1 - splunk-system-user [20/Feb/2025:08:40:34.605 +0100] "GET /servicesNS/nobody/alert_manager_enterprise/messages/ame-index-resilience-default-error HTTP/1.0" 404 177 "-" "splunk-sdk-python/1.7.3" - - - 0ms
0 Karma

Andras
Loves-to-Learn

Hi @seiimonn !

Debian GNU/Linux 12 (bookworm)

Splunk Enterprise 9.0.0

AME 3.0.8.

Sysinfo:

{"uuid":"95c6740c-9e0b-42b1-b2b9-b78067db6677","status":200,"messages":[],"payload":{"tenant_list":[{"tenant_uid":"default","role":"admin"}],"is_admin":true,"is_app_admin":true,"products":[],"necessary_tasks":[],"legacy_installed":false,"environment":"on_premises","timezone":"UTC"}}

There ara no errors now, if i run this script:

index=_internal source=*ame* ERROR | table _time host source _raw


But maybe these ara interesting in the splunkd.log:

19/02/2025
19:34:15.506
 
02-19-2025 19:34:15.506 +0100 WARN HttpListener [1069 HttpDedicatedIoThread-4] - Socket error from 127.0.0.1:37790 while accessing /servicesNS/nobody/alert_manager_enterprise/properties/server: Broken pipe
 19/02/2025
19:34:08.828
 
2025-02-19 19:34:08,828 INFO [assist::supervisor_modular_input.py] [context] [build_supervisor_secrets] [22691] Secret load failed, key=tenant_id, error=[HTTP 404] https://127.0.0.1:8089/servicesNS/nobody/splunk_assist/storage/passwords/tenant_id?output_mode=json
 19/02/2025
19:34:06.362
 
02-19-2025 19:34:06.362 +0100 WARN HttpListener [1068 HttpDedicatedIoThread-3] - Socket error from 127.0.0.1:52422 while accessing /servicesNS/nobody/alert_manager_enterprise/properties/server: Broken pipe
 19/02/2025
19:33:56.500
 
2025-02-19 19:33:56.500 +0100 Trace-Id= type=METER, name=ch.qos.logback.core.Appender.error, count=3, m1_rate=3.527460396057507E-12, m5_rate=9.325633072421824E-5, m15_rate=7.016228689718483E-4, mean_rate=0.0019981503731937404, rate_unit=events/second
 19/02/2025
19:33:54.415
 
2025-02-19 19:33:54,415 INFO [assist::supervisor_modular_input.py] [context] [build_supervisor_secrets] [22467] Secret load failed, key=tenant_id, error=[HTTP 404] https://127.0.0.1:8089/servicesNS/nobody/splunk_assist/storage/passwords/tenant_id?output_mode=json

 

I use this script, to create test alerts:

| makeresults | eval user="World", src="192.168.0.1", action="create test event" | sendalert create_alert param.title="Hello $result.user$" param.template=default param.tenant_uid=default

I think there is nothing interesting on the browsers developer console. What do you think about that?

Thanks for your helping.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Andras ,

you can see in Alert Manager App only alerts share at Global level, so you have to change the permissions in your alerts from App level to Global level.

Ciao.

Giuseppe

0 Karma

Andras
Loves-to-Learn

Hi @gcusello !

Also interesting that the alerts in the index seems good:

Andras_0-1739974989861.png

But the loading of the events in the Events dashboard never ending.

Andras_1-1739975075931.png

 

 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Andras ,

if you haven't the issue about the Global Sharing of the Alerts, check the macros used in the dashboards, probably you have to specify the index where Notables are located.

You can do it opening the dashboard.

Ciao.

Giuseppe

0 Karma

Andras
Loves-to-Learn

Hi Giuseppe!

Thank you for your answer.

I double checked, but my alerts are alredy global.

I think there is another problem.

Thanks,

A

0 Karma
Get Updates on the Splunk Community!

Splunk App Dev Community Updates – What’s New and What’s Next

Welcome to your go-to roundup of everything happening in the Splunk App Dev Community! Whether you're building ...

The Latest Cisco Integrations With Splunk Platform!

Join us for an exciting tech talk where we’ll explore the latest integrations in Cisco + Splunk! We’ve ...

Enterprise Security Content Update (ESCU) | New Releases

In April, the Splunk Threat Research Team had 2 releases of new security content via the Enterprise Security ...