All Apps and Add-ons

Microsoft Teams alert action not working for some alerts, but is for others

mmccul_fe
Explorer

Trying to configure various alerts to use Microsoft Teams. For one alert, it works reliably, each time showing up. Other alerts, I get no notice at all.

Overall log of a failed attempt to send an alert according to _internal (anonymized):

06-27-2019 19:33:06.721 +0000 INFO  Metrics - group=per_source_thruput, series="/opt/splunk/var/log/splunk/microsoft_teams_webhook_modalert.log", kbps=0.049490623625275856, eps=0.16129343918613137, kb=1.5341796875, ev=5, avg_age=0, max_age=0
06-27-2019 19:33:03.752 +0000 ERROR SearchScheduler - Error in 'sendalert' command: Alert script returned error code 4., search='sendalert microsoft_teams_webhook results_file="/opt/splunk/var/run/splunk/dispatch/scheduler__nobody_c3BsdW5rX21vbml0b3JpbmdfY29uc29sZQ__RMD544e6bdae8b4cae07_at_1561663980_61535/results.csv.gz" results_link="https://host.example.com/en-US/app/splunk_monitoring_console/@go?sid=scheduler__nobody_c3BsdW5rX21vbml0b3JpbmdfY29uc29sZQ__RMD544e6bdae8b4cae07_at_1561663980_61535"'
06-27-2019 19:33:03.751 +0000 WARN  sendmodalert - action=microsoft_teams_webhook - Alert action script returned error code=4
06-27-2019 19:33:03.751 +0000 INFO  sendmodalert - action=microsoft_teams_webhook - Alert action script completed in duration=2277 ms with exit code=4
2019-06-27 19:33:03,737 ERROR pid=30021 tid=MainThread file=cim_actions.py:message:238 | sendmodaction - signature="Error: 'NoneType' object has no attribute 'split'. Please double check spelling and also verify that a compatible version of Splunk_SA_CIM is installed." action_name="microsoft_teams_webhook" search_name="DMC Alert - Total License Usage Near Daily Quota" sid="scheduler__nobody_c3BsdW5rX21vbml0b3JpbmdfY29uc29sZQ__RMD544e6bdae8b4cae07_at_1561663980_61535" rid="0" app="splunk_monitoring_console" user="nobody" action_mode="saved" action_status="failure"
2019-06-27 19:33:03,737 INFO pid=30021 tid=MainThread file=cim_actions.py:message:238 | sendmodaction - signature="Alert action microsoft_teams_webhook started." action_name="microsoft_teams_webhook" search_name="DMC Alert - Total License Usage Near Daily Quota" sid="scheduler__nobody_c3BsdW5rX21vbml0b3JpbmdfY29uc29sZQ__RMD544e6bdae8b4cae07_at_1561663980_61535" rid="0" app="splunk_monitoring_console" user="nobody" action_mode="saved" action_status="success"
2019-06-27 19:33:01,748 INFO pid=30021 tid=MainThread file=cim_actions.py:message:238 | sendmodaction - signature="Invoking modular action" action_name="microsoft_teams_webhook" search_name="DMC Alert - Total License Usage Near Daily Quota" sid="scheduler__nobody_c3BsdW5rX21vbml0b3JpbmdfY29uc29sZQ__RMD544e6bdae8b4cae07_at_1561663980_61535" rid="0" app="splunk_monitoring_console" user="nobody" action_mode="saved"
06-27-2019 19:33:01.473 +0000 INFO  sendmodalert - Invoking modular alert action=microsoft_teams_webhook for search="DMC Alert - Total License Usage Near Daily Quota" sid="scheduler__nobody_c3BsdW5rX21vbml0b3JpbmdfY29uc29sZQ__RMD544e6bdae8b4cae07_at_1561663980_61535" in app="splunk_monitoring_console" owner="nobody" type="saved"

Any suggestions as to what might be wrong?

I've rechecked the submit URL to confirm it is identical between the one that works and the ones that do not.

0 Karma

skazako
New Member

A workaround could be sending an email to a specific MS Teams channel.  
Just get its email address by going to the channel name and click More options  > Get email address.
https://www.youtube.com/watch?v=d5Dekg8NG5w

0 Karma

davidcottingham
Explorer

I believe that in order for the script to post output you need to use | Table, or the request format is not valid. There was a great blog post on it by Lisa Rushworth here https://www.rushworth.us/lisa/?tag=splunk

I have taken some of her suggestions, added in proxy support and compiled it in another version of this app that works for all outputs here: https://github.com/cottinghamd/Splunk-Microsoft-Teams-Webhook-Connector

mmccul_fe
Explorer

Unfortunately, the built in MC alerts are some of the ones not functioning, and when I added an alert to the search head that had an explicit table as the last statement of the query, it is one of the ones not working at all.

index=_internal KEYWORD_HERE source="/opt/splunk/var/log/splunk/ta_obfuscated_here.log" HTTPError |
rex "HTTPError:\s*(?<status_code>\d{3}).+(?<url>https?://\S+)" |
table host,status_code,url

(Try not to laugh at my search, it works)

I'll check out your alternate version, see if I can get that to work.

0 Karma
Get Updates on the Splunk Community!

Splunk Education - Fast Start Program!

Welcome to Splunk Education! Splunk training programs are designed to enable you to get started quickly and ...

Five Subtly Different Ways of Adding Manual Instrumentation in Java

You can find the code of this example on GitHub here. Please feel free to star the repository to keep in ...

New Splunk APM Enhancements Help Troubleshoot Your MySQL and NoSQL Databases Faster

Splunk Observability has two new enhancements to make it quicker and easier to troubleshoot slow or frequently ...