All Apps and Add-ons

Microsoft Teams Add-on for Splunk: handling of 404 error

Path Finder

I downloaded and installed Teams Add-on for Splunk and it worked for a while, until we encountered a lot of 404 error like below

ERROR pid=14248 tid=MainThread | Error getting callRecord data: 404 Client Error: Not Found for url:<call ID>?$expand=sessions($expand=segments)


I found out that the callID was removed from Teams CDR for some reason,  therefore when Splunk tried to download the CDR, it returned error 404, which is understandable.

However Teams Add-on will not remove the Call ID from webhook directory for this scenario. The call ID will remain there forever and Splunk will keep on trying again and again to download the CDR and failed. This results in a huge amount of call IDs that never get cleaned up and massive number of error messages in the log.

Further more, i found out that if there were too many call ID files exist in the wehbook directory (~60K), the Add-on will encountered error "401 Unauthorized to download the CDR" and stopped working soon afterward. After restarting Splunk, the Add-on worked again and then stopped the  moment it hit 401 error again. I manually created a script to manage the load of webhook folder, so this is OK for now but it would be preferable that the Add-on has load management feature by itself.

Hopefully the author of this Add-on will add this error handling soon, but meanwhile if anyone knows how to get around this 404 issue please kindly share.

Thanks a lot!

Labels (2)
0 Karma

Splunk Employee
Splunk Employee



The reason for that is the call ID is no longer available at Azure side, but MS Teams addon tries to get information with it.

Currently there is no way except to delete the local kv store lookup data.


Please try the below command and see if there is any improvements.


splunk clean kvstore -app TA_MS_Teams -collection TA_MS_Teams_checkpointer


0 Karma

Path Finder

I dont think this has anything to do with kvstore.

problem is clear: the add-on doesn't handle 404 error properly

Flow in normal situation:

check webhook folder for call ID --> download call ID --> delete call ID from webhook folder --> proceed with next ID

Flow in 404 situation:

check webhook folder for call ID --> download call ID (but failed) --> raise error.  And it stops there, the call ID with 404 error is not cleaned up from webhook folder.

So i think the author of the app just needs to improve the error handling to clean up "404" call ID from the webhook folder, problem will be solved

0 Karma


i totally agree.  This app does not handle 400 or 404 errors.  The developer  @jconger   is top notch though, i have met him.  just this one app have never really worked correctly

We have to reset the inputs almost daily  By reset, we create a new subscription input.  Or we have to disable/reenable the call record or user report inputs

However, i did perform the kvstore clean on both of our heavy fowarders (behind Load balancer)

for a load balancer environment webhook, call record, and user report inputs are setup on both HF, but subscription is setup on only one

This worked for me, for now

disable all inputs 

clean kvstore

splunk clean kvstore -app TA_MS_Teams -collection TA_MS_Teams_checkpointer

enable inputs in this order

webhook, subscription, call record, user report


we have data again...for now. 







0 Karma