Hi @Cbr1sg , Based on the thread and my experience with this issue, the Microsoft Teams Add-on for Splunk has a known deficiency in handling 404 errors properly. Here's what's happening: 1. When the add-on encounters call IDs that no longer exist in Microsoft Teams (returning 404 errors), it fails to remove these IDs from the webhook directory. 2. This causes a build-up of unprocessable call IDs, leading to: a. Continuous error messages in the logs b. Eventually a "401 Unauthorized" error when too many files accumulate (~60K files) c. The add-on completely stops working until restarted The most reliable fix I've found is the following procedure that needs to be performed periodically (some users report every few days): ## Solution Steps: 1. Disable all inputs in this order: a. Call Record input b. User Report input (if configured) c. Subscription input d. Webhook input 2. Clean the KVStore to reset the checkpointer: splunk clean kvstore -app TA_MS_Teams -collection TA_MS_Teams_checkpointer Note: You need to run this command on the machine where the add-on is installed (usually a heavy forwarder). 3. Re-enable the inputs in this specific order: a. Webhook input b. Subscription input (this will recreate the subscription) c. Call Record input d. User Report input (if used) 4. Additional steps for persistent solution: a. If you're comfortable with scripting, you can create a scheduled task (cron job) to run these steps nightly b. For a more advanced solution, you could create an alert that triggers when "404 Not Found" errors appear in logs ## Scripted Solution Example: Here's a bash script that you could schedule to run nightly: bash #!/bin/bash # Path to Splunk binary SPLUNK_BIN="/opt/splunk/bin/splunk" # Disable inputs $SPLUNK_BIN disable input TA_MS_Teams://call_record $SPLUNK_BIN disable input TA_MS_Teams://user_report $SPLUNK_BIN disable input TA_MS_Teams://subscription $SPLUNK_BIN disable input TA_MS_Teams://webhook # Wait for processes to stop sleep 10 # Clean KVStore $SPLUNK_BIN clean kvstore -app TA_MS_Teams -collection TA_MS_Teams_checkpointer # Re-enable inputs in correct order $SPLUNK_BIN enable input TA_MS_Teams://webhook sleep 5 $SPLUNK_BIN enable input TA_MS_Teams://subscription sleep 10 $SPLUNK_BIN enable input TA_MS_Teams://call_record $SPLUNK_BIN enable input TA_MS_Teams://user_report echo "Microsoft Teams Add-on inputs reset completed at $(date)" Note that while this is a functional workaround, the root issue is in the add-on's code not properly handling 404 errors. As mentioned by others in the thread, the add-on should ideally be updated to remove call IDs from the webhook directory when they return 404 errors. If you're experiencing this issue frequently, I recommend also opening a support case with Splunk to encourage the development team to address this in a future update of the add-on. Please give 👍 for support 😁 happly splunking .... 😎
... View more