topic Re: Splunk Alerts not sending e-mail in Alerting

Splunk Alerts not sending e-mail

golcondar — Fri, 06 Mar 2020 02:06:43 GMT

Hi,

I've created a Splunk alert (see below photos) and have found that it's not properly sending e-mails to my account upon being triggered.
I opened the query in the search bar (from the alerts page) to verify that the message i'm looking for is actually showing up, which it is.

I'm not sure what the problem might be.
Please let me know if there's any other information which I could include that might be helpful.

Thanks!

Re: Splunk Alerts not sending e-mail

gcusello — Fri, 06 Mar 2020 13:46:43 GMT

Hi @golcondar,
I think that you already configured Splunk to send eMails and that there are other alerts that correctly run.

At first, check if the dimension of the pdf exceed the limit of your eMail attachement.
Then you can see in _internal, if there's some event related.

Ciao.
Giuseppe

Re: Splunk Alerts not sending e-mail

woodcock — Fri, 06 Mar 2020 16:41:03 GMT

First, try sending ad-hoc by using the | sendemail command in your SPL. Then check here:

index=_* AND (SMTP OR sendemail OR email) AND (FAIL* OR ERR* OR TIMEOUT OR CANNOT OR REFUSED OR REJECTED)

Re: Splunk Alerts not sending e-mail

golcondar — Fri, 06 Mar 2020 22:09:29 GMT

Hi,

I was able to get e-mail results by using |sendemail.
However, my alert still did not trigger, and I also put in the query you placed above and got no results.
I'll attach some images to the next post (it's not letting me attach them to this one) to show what i did.

Re: Splunk Alerts not sending e-mail

golcondar — Fri, 06 Mar 2020 22:13:49 GMT

Looks like I can't add any more images to this post 😞
I took the query from my alert and added the |sendemail command to the end, so I know that the query itself is correct.

I entered the below to search for errors:

index=_* AND (SMTP OR sendemail OR email) AND (FAIL* OR ERR* OR TIMEOUT OR CANNOT OR REFUSED OR REJECTED)

but got no results.
Any ideas on what I could do next?

Re: Splunk Alerts not sending e-mail

golcondar — Fri, 06 Mar 2020 22:15:05 GMT

Hi,

I don't need the PDF attachment so I went ahead and deselected it. That didn't end up fixing the issue. I also wasn't able to get any results from searching _internal unfortunately.

Re: Splunk Alerts not sending e-mail

woodcock — Fri, 06 Mar 2020 23:01:29 GMT

I have seen this happen before where people are expecting for an email to ALWAYS be sent when something fails but they have the alert set with:
Trigger alert when = Number of Results with is equal to and 0 combined with
Trigger = For each result
The solution is to set Trigger = Once. If you stop and think about it, it makes TOTAL sense why it doesn't send the email.

In your case, because you have an older version of Splunk, the GUI is a bit different; you need to click on Per-Result and choose the other option, which I believe is Digest.

Re: Splunk Alerts not sending e-mail

golcondar — Sat, 07 Mar 2020 00:14:44 GMT

I'll try swapping it to "Per Result" instead of what I currently have and seeing if that works; i recall attempting that before and still not getting the e-mails.
If it still doesn't work, i'll attempt it with Trigger=Once.

Re: Splunk Alerts not sending e-mail

golcondar — Mon, 09 Mar 2020 22:55:33 GMT

Looks like it was an issue with permissions. I had a coworker who had created alerts before successfully follow the same process as me and the alert properly sent the e-mail.

Thanks for the help everyone!