Alerting

troubleshooting email alerts in splunk

mburgess97
Path Finder

I have several alerts that have been firing off an email. Everything has been working for several weeks. However, I noticed over the weekend that search results that should have triggered did not send an email. I don't see any errors in the scheduler.log file.

Any other recommendations for troubleshooting this issue?

0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

Look in the jobs log (Activity->Jobs) to see if the jobs ran then check the alerts log (Activity->Triggered Alerts) to see if they triggered alerts. Then check splunkd.log for "sendemail" to see if there were problems sending mail.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

nvanderwalt_spl
Splunk Employee
Splunk Employee

When you are looking at historic data, you may see something that is there now, but may not have made it in to the system in time when when the job ran because of indexing lag

index=searched index host=host_that_did_not_get_triggered  other relevant filters
| eval lagSecs = _indextime - _time 
| timechart avg(lagSecs)

If you are unable to see any results for this job, then perhaps the search has not run in a while. You did mention you did not see any errors in scheduler log. Is this search actually running when scheduled?

richgalloway
SplunkTrust
SplunkTrust

Look in the jobs log (Activity->Jobs) to see if the jobs ran then check the alerts log (Activity->Triggered Alerts) to see if they triggered alerts. Then check splunkd.log for "sendemail" to see if there were problems sending mail.

---
If this reply helps you, Karma would be appreciated.

mburgess97
Path Finder

I can't see past 06-23 - I have no idea why I wouldn't be seeing logs beyond that date.

So,

Activity -> Jobs (Nothing past 6/23 though I've been running this for weeks and it's been working)
Activity -> Triggered Alerts (empty - though I think this is due to older than 6/23)
Splunkd (zero sendemail logs in the file)

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Using the Splunk Threat Research Team’s Latest Security Content

REGISTER HERE Tech Talk | Security Edition Did you know the Splunk Threat Research Team regularly releases ...

SplunkTrust | 2024 SplunkTrust Application Period is Open!

It's that time again, folks! That's right, the application/nomination period for the 2024 SplunkTrust is ...