Why are Splunk alerts not sending to cell phone/mobile addresses?

[weisenbm]

I have an alert configured in Splunk that, when triggered, I'd like it to send a message to an email address along with some of my team members cell phones. The alert appears to be configured correctly: the alert is triggered when the correct conditions are met, and the email is sent to the correct email address, However, the message is not sent to the cell phones of me or my team. We are trying to understand why this is happening.

There are a few additional things to note that may help guide any advice you can offer; one is that the phone numbers listed in the Splunk alert have the correct email notation i.e. a Verizon number has phoneNumber@vtext.com and so on.

Secondly is that this alert has previously sent messages to our phone correctly, and we can not diagnose the reason why that may have changed and we are no longer receiving it.

Lastly, I mentioned earlier that this problem only exists with the 'phone-email' addresses, and that the alert message DOES get sent to an Outlook email address. Well, when I check that Outlook email, the 'phone-email' addresses of me and my team members are listed as recipients of the email, but we are not receiving the text alerts.

Please, any help, advice, or things to check into are greatly appreciated. Thanks!

[woodcock]

Do a search like this:

index=_* phoneNumber@vtext.com (err* OR warn* OR cannot* OR fail* OR timeout OR reject* OR deni* OR deny)

Also, check your gateway's reputation for blacklisting here:
https://mxtoolbox.com/blacklists.aspx
Lastly, be sure to talk to your email admin. He will surely be able to get to the bottom of this.

[cpetterborg]

Also, if the length of the message is too big, that can cause it to be rejected for an SMS message. Has the length of the message increased?

[delappml3]

Based on the information you provided, it seems that Splunk is passing the alert email messages on to your organization's SMTP server just fine. Is it possible that the SMTP server or email infrastructure is enforcing a policy to block or filter applications from sending email to external addresses? It seems that the main difference at first glance is that the successfully received email alerts are sent to internal email addresses, while external email recipients do not receive them. If this is indeed the situation, you might want to talk with your email team to see if a rule can be changed in their system to allow sending of messages from Splunk to external addresses.

[richgalloway]

What errors are in the search log or in splunkd.log? There should be something after "sendemail".

[efavreau]

If the logs don't help, trace the entire pipeline. Did you upgrade Splunk? Did you upgrade sendmail or whatever on the Splunk servers? Did you update firewalls? Did you update router/load balancer/traffic mgr software/hardware/rules? Can you email the phone addresses from your outlook client on the same network as the Splunk servers? Create a map of every single device between your Splunk server(s) and the internet responsible for the network and software and hardware to get an email out. Test every single node also. So if you have 4 splunk servers that could send the email, test each one individually. All of this information may lead to your solution, and if not, you have a lot more data to give back to us.