Alerting

Splunk Cloud - Why are we not receiving Splunk alert emails?

jackin
Path Finder

Hi Team,

We are unable to get the alert emails even when the events matching the alert condition is present in Splunk cloud.

Please help how we can resolve this?

Labels (1)
0 Karma

madhav_dholakia
Contributor

Hi, we are also facing the same issue since this morning around 11 AM BST. No scheduled alert/report emails are not being sent. also tried the test email but it didn't work as well. thank you.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Search splunkd.log for "sendemail" to see if Splunk is reporting errors sending email.  If not then your email provider may be discarding the messages as spam.  Contact them.

---
If this reply helps you, Karma would be appreciated.
0 Karma

madhav_dholakia
Contributor

Hi @richgalloway 

tried sending an email using this search but no luck.

index=main | head 5 | sendemail to="firstname.lastname@email.address" server="localhost" subject="Test Mail" message="This is an example message" sendresults=true inline=true format=raw sendpdf=true


Also, when checking below search, it doesn't show any errors:

index="_internal" source="/opt/splunk/var/log/splunk/python.log" sendemail


INFO sendemail:184 - Sending email
INFO sendemail:1516 - Generated PDF for email

we have tried checking the emails for two different domains (where we have received emails until yesterday) and no issues with email blocking/black listing.

can you please suggest what else could be checked?

Thank you.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Something changed yesterday to prevent Splunk Cloud emails from being delivered.  I suggest open a Support Request to have Splunk check things on their end and also working with your network team to verify Splunk Cloud email is allowed in.

---
If this reply helps you, Karma would be appreciated.
0 Karma

madhav_dholakia
Contributor

thanks, @richgalloway - We have raised a ticket with support for this - We also checked with internal it team and no emails were blocked - also the emails were not received by other domain as well.  these emails were not received for almost 4 hours and then without any actions, it started.  

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

this sounds like there are one or more mail servers between SC and your mail servers which have some issues and cannot deliver mails online. They just queued those and send those later on when temporary resource issues have fixed. In old days that was quite common situation when servers and users has more limited quotas etc.

r. Ismo

0 Karma

madhav_dholakia
Contributor

Hi @isoutamo 

If it was a case of queueing, we would have received all the hourly email alerts once the temp resource issues were fixed – but it was not the case – we completely missed the emails (and not received lately).

Also, there were different email servers affected and only Splunk emails were not being received – without making any changes on these email servers, we started receiving emails at around 15:00 BST – the issue was only observed between 11:00-15:00 BST on 31st Jul.

Do you suggest if anything more specific that I should check?

Thank you.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...