Alerting

Why were my triggered alerts cleared after restart?

ejread
Explorer

While working through 10+ triggered alerts shown on the "Triggered Alerts" page, we had to restart the search head for maintenance. After the restart finished, all the triggered alerts disappeared. Is this expected or not?

Tags (2)
1 Solution

gfuente
Motivator

This is the expected behaviour. The fired alerts are search artifacts that persist in the var (temporal) folder. So whenSplunk is restarted those artifacts are deleted.

I would recommend you using this app, to improve the splunk default alerting system:

https://splunkbase.splunk.com/app/2665/

It stores the alerts as incidents in the KV store, so they will survive to restarts

Regards

View solution in original post

gfuente
Motivator

This is the expected behaviour. The fired alerts are search artifacts that persist in the var (temporal) folder. So whenSplunk is restarted those artifacts are deleted.

I would recommend you using this app, to improve the splunk default alerting system:

https://splunkbase.splunk.com/app/2665/

It stores the alerts as incidents in the KV store, so they will survive to restarts

Regards

joshua_hart1
Path Finder

Thanks. I have Alert Manager; though it's in a testing phase right now. We'll look at that for a more permanent solution. Thanks again.

  • Josh
0 Karma

joshua_hart1
Path Finder

Did you ever figure this one out? We're using REST to populate a dashboard with fired alerts and noticed the same behavior.

0 Karma

neelamssantosh
Contributor

Are you restarting the server as its getting hanged or due to heavy load issues.
And i hope, Unfortunately it would be yes as after restart your Dispatch/cache/buffer will get cleared as the OS.

0 Karma

joshua_hart1
Path Finder

Splunk service is being restart in order to read in new configs and install new apps. It doesn't seem smart that the alerts are stored on disk (kv store, summary index, etc.)

0 Karma
Get Updates on the Splunk Community!

Splunk Education - Fast Start Program!

Welcome to Splunk Education! Splunk training programs are designed to enable you to get started quickly and ...

Five Subtly Different Ways of Adding Manual Instrumentation in Java

You can find the code of this example on GitHub here. Please feel free to star the repository to keep in ...

New Splunk APM Enhancements Help Troubleshoot Your MySQL and NoSQL Databases Faster

Splunk Observability has two new enhancements to make it quicker and easier to troubleshoot slow or frequently ...