Alerting

troubleshooting email alerts in splunk

mburgess97
Path Finder

I have several alerts that have been firing off an email. Everything has been working for several weeks. However, I noticed over the weekend that search results that should have triggered did not send an email. I don't see any errors in the scheduler.log file.

Any other recommendations for troubleshooting this issue?

0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

Look in the jobs log (Activity->Jobs) to see if the jobs ran then check the alerts log (Activity->Triggered Alerts) to see if they triggered alerts. Then check splunkd.log for "sendemail" to see if there were problems sending mail.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

nvanderwalt_spl
Splunk Employee
Splunk Employee

When you are looking at historic data, you may see something that is there now, but may not have made it in to the system in time when when the job ran because of indexing lag

index=searched index host=host_that_did_not_get_triggered  other relevant filters
| eval lagSecs = _indextime - _time 
| timechart avg(lagSecs)

If you are unable to see any results for this job, then perhaps the search has not run in a while. You did mention you did not see any errors in scheduler log. Is this search actually running when scheduled?

richgalloway
SplunkTrust
SplunkTrust

Look in the jobs log (Activity->Jobs) to see if the jobs ran then check the alerts log (Activity->Triggered Alerts) to see if they triggered alerts. Then check splunkd.log for "sendemail" to see if there were problems sending mail.

---
If this reply helps you, Karma would be appreciated.

mburgess97
Path Finder

I can't see past 06-23 - I have no idea why I wouldn't be seeing logs beyond that date.

So,

Activity -> Jobs (Nothing past 6/23 though I've been running this for weeks and it's been working)
Activity -> Triggered Alerts (empty - though I think this is due to older than 6/23)
Splunkd (zero sendemail logs in the file)

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...