Alerting

Why are email alerts not sent until Splunk restart?

Communicator

We have had a problem over the weekend when one of our alerts did not trigger. I had to restart the services to get it all working again.
Does anyone had any idea why this might have happened?
It's possible it was related to changes we had made. It's the second time in a week we've needed to restart the services to have changes start working.

0 Karma
1 Solution

Communicator

In the end it appeared that the splunk server was skipping triggering as apparently there is a limit to 1 real time alert per CPU core.
We increased this and it mostly fixed the issues.

View solution in original post

0 Karma

Communicator

In the end it appeared that the splunk server was skipping triggering as apparently there is a limit to 1 real time alert per CPU core.
We increased this and it mostly fixed the issues.

View solution in original post

0 Karma

Revered Legend

What changes did you make and how (deployed/updated conf files/from UI)?

0 Karma

Communicator

I create a new lookup table, added the new fields to search and also to the email alert that went out.

0 Karma

Communicator

I should say i tested it and it was working 11.55pm on friday. Then nothing for the rest of the weekend

0 Karma

Revered Legend

Did you check the scheduler logs for whether the alert search was run and if there were results that would trigger the alert?

index=_internal sourcetype=scheduler savedsearch_name="YourAlertName"
0 Karma

Communicator

Thanks for this, yes it shows that it ran 1441 times on that day. Meaning it ran every minuet in the day so all working well..

Also if i run the search that the alert is built on the event shows up so i know the criteria was met.

This morning i sent a test alert no email
restarted services
sent another test alert and it worked.

0 Karma
Don’t Miss Global Splunk
User Groups Week!

Free LIVE events worldwide 2/8-2/12
Connect, learn, and collect rad prizes
and swag!