Microsoft Teams alert action not working for some alerts, but is for others

[mmccul_fe]

Trying to configure various alerts to use Microsoft Teams. For one alert, it works reliably, each time showing up. Other alerts, I get no notice at all.

Overall log of a failed attempt to send an alert according to _internal (anonymized):

06-27-2019 19:33:06.721 +0000 INFO  Metrics - group=per_source_thruput, series="/opt/splunk/var/log/splunk/microsoft_teams_webhook_modalert.log", kbps=0.049490623625275856, eps=0.16129343918613137, kb=1.5341796875, ev=5, avg_age=0, max_age=0
06-27-2019 19:33:03.752 +0000 ERROR SearchScheduler - Error in 'sendalert' command: Alert script returned error code 4., search='sendalert microsoft_teams_webhook results_file="/opt/splunk/var/run/splunk/dispatch/scheduler__nobody_c3BsdW5rX21vbml0b3JpbmdfY29uc29sZQ__RMD544e6bdae8b4cae07_at_1561663980_61535/results.csv.gz" results_link="https://host.example.com/en-US/app/splunk_monitoring_console/@go?sid=scheduler__nobody_c3BsdW5rX21vbml0b3JpbmdfY29uc29sZQ__RMD544e6bdae8b4cae07_at_1561663980_61535"'
06-27-2019 19:33:03.751 +0000 WARN  sendmodalert - action=microsoft_teams_webhook - Alert action script returned error code=4
06-27-2019 19:33:03.751 +0000 INFO  sendmodalert - action=microsoft_teams_webhook - Alert action script completed in duration=2277 ms with exit code=4
2019-06-27 19:33:03,737 ERROR pid=30021 tid=MainThread file=cim_actions.py:message:238 | sendmodaction - signature="Error: 'NoneType' object has no attribute 'split'. Please double check spelling and also verify that a compatible version of Splunk_SA_CIM is installed." action_name="microsoft_teams_webhook" search_name="DMC Alert - Total License Usage Near Daily Quota" sid="scheduler__nobody_c3BsdW5rX21vbml0b3JpbmdfY29uc29sZQ__RMD544e6bdae8b4cae07_at_1561663980_61535" rid="0" app="splunk_monitoring_console" user="nobody" action_mode="saved" action_status="failure"
2019-06-27 19:33:03,737 INFO pid=30021 tid=MainThread file=cim_actions.py:message:238 | sendmodaction - signature="Alert action microsoft_teams_webhook started." action_name="microsoft_teams_webhook" search_name="DMC Alert - Total License Usage Near Daily Quota" sid="scheduler__nobody_c3BsdW5rX21vbml0b3JpbmdfY29uc29sZQ__RMD544e6bdae8b4cae07_at_1561663980_61535" rid="0" app="splunk_monitoring_console" user="nobody" action_mode="saved" action_status="success"
2019-06-27 19:33:01,748 INFO pid=30021 tid=MainThread file=cim_actions.py:message:238 | sendmodaction - signature="Invoking modular action" action_name="microsoft_teams_webhook" search_name="DMC Alert - Total License Usage Near Daily Quota" sid="scheduler__nobody_c3BsdW5rX21vbml0b3JpbmdfY29uc29sZQ__RMD544e6bdae8b4cae07_at_1561663980_61535" rid="0" app="splunk_monitoring_console" user="nobody" action_mode="saved"
06-27-2019 19:33:01.473 +0000 INFO  sendmodalert - Invoking modular alert action=microsoft_teams_webhook for search="DMC Alert - Total License Usage Near Daily Quota" sid="scheduler__nobody_c3BsdW5rX21vbml0b3JpbmdfY29uc29sZQ__RMD544e6bdae8b4cae07_at_1561663980_61535" in app="splunk_monitoring_console" owner="nobody" type="saved"

Any suggestions as to what might be wrong?

I've rechecked the submit URL to confirm it is identical between the one that works and the ones that do not.

[skazako]

A workaround could be sending an email to a specific MS Teams channel.  
Just get its email address by going to the channel name and click More options  > Get email address.
https://www.youtube.com/watch?v=d5Dekg8NG5w

[davidcottingham]

I believe that in order for the script to post output you need to use | Table, or the request format is not valid. There was a great blog post on it by Lisa Rushworth here https://www.rushworth.us/lisa/?tag=splunk

I have taken some of her suggestions, added in proxy support and compiled it in another version of this app that works for all outputs here: https://github.com/cottinghamd/Splunk-Microsoft-Teams-Webhook-Connector

[mmccul_fe]

Unfortunately, the built in MC alerts are some of the ones not functioning, and when I added an alert to the search head that had an explicit table as the last statement of the query, it is one of the ones not working at all.

index=_internal KEYWORD_HERE source="/opt/splunk/var/log/splunk/ta_obfuscated_here.log" HTTPError |
rex "HTTPError:\s*(?<status_code>\d{3}).+(?<url>https?://\S+)" |
table host,status_code,url

(Try not to laugh at my search, it works)

I'll check out your alternate version, see if I can get that to work.