Greetings folks.
I installed the TA-ms-teams-alert-action to... you probably guessed... send alert messages to Teams. After installation exactly two messages were sent successfully to Teams. I even took screenshots. I recently realized I had not received any messages for events that I knew had happened so I started digging. Looks like a lot of messages are stuck in a resending state.
Further digging in the logs indicates that when the TA tried to send a message to the Teams webhook it received a 404:
2022-04-06 00:35:45,922 ERROR pid=123018 tid=MainThread file=cim_actions.py:message:280 | sendmodaction - signature="Microsoft Teams publish to channel has failed!. url=https://totallyvalid.webhook.office.com/webhookb2/XXXXX , data={
}, HTTP Error=404, HTTP Reason=Not Found, HTTP content=<!DOCTYPE html>
<span><H1>Server Error in '/WebhookB2' Application.<hr width=100% size=1 color=silver></H1>
<h2> <i>The resource cannot be found.</i> </h2></span>
<font face="Arial, Helvetica, Geneva, SunSans-Regular, sans-serif ">
<b> Description: </b>HTTP 404. The resource you are looking for (or one of its dependencies) could have been removed, had its name changed, or is temporarily unavailable. Please review the following URL and make sure that it is spelled correctly.
<br><br>
<b> Requested URL: </b>/webhookb2/XXXXX<br><br>
I am unclear how to proceed. I've changed the web hook URLs above for privacy but the hooks in the logs and in the TA match the hooks in the Teams connector configuration. I know the webhooks work because they are in use by other tools and are not failing.
I tested the webhooks from my laptop and was able to send a message. I tested the webhook from a search head and was able to send a message. Something appears to be munging the web hook URL but I cannot determine how or where. And since it worked previously and has not changed (I am the only person with access) I can't figure it out. I suspect that this line "Server Error in '/WebhookB2' Application." is relevant.
This is on Splunk Enterprise 8.2.2.2.
Thoughts or strategies would be appreciated.
