Alerting

Why are email alerts not sent until Splunk restart?

arrowecssupport
Communicator

We have had a problem over the weekend when one of our alerts did not trigger. I had to restart the services to get it all working again.
Does anyone had any idea why this might have happened?
It's possible it was related to changes we had made. It's the second time in a week we've needed to restart the services to have changes start working.

0 Karma
1 Solution

arrowecssupport
Communicator

In the end it appeared that the splunk server was skipping triggering as apparently there is a limit to 1 real time alert per CPU core.
We increased this and it mostly fixed the issues.

View solution in original post

0 Karma

arrowecssupport
Communicator

In the end it appeared that the splunk server was skipping triggering as apparently there is a limit to 1 real time alert per CPU core.
We increased this and it mostly fixed the issues.

View solution in original post

0 Karma

somesoni2
Revered Legend

What changes did you make and how (deployed/updated conf files/from UI)?

0 Karma

arrowecssupport
Communicator

I create a new lookup table, added the new fields to search and also to the email alert that went out.

0 Karma

arrowecssupport
Communicator

I should say i tested it and it was working 11.55pm on friday. Then nothing for the rest of the weekend

0 Karma

somesoni2
Revered Legend

Did you check the scheduler logs for whether the alert search was run and if there were results that would trigger the alert?

index=_internal sourcetype=scheduler savedsearch_name="YourAlertName"
0 Karma

arrowecssupport
Communicator

Thanks for this, yes it shows that it ran 1441 times on that day. Meaning it ran every minuet in the day so all working well..

Also if i run the search that the alert is built on the event shows up so i know the criteria was met.

This morning i sent a test alert no email
restarted services
sent another test alert and it worked.

0 Karma