All Apps and Add-ons

Microsoft Teams Add-on for Splunk: handling of 404 error

Cbr1sg
Path Finder

I downloaded and installed Teams Add-on for Splunk and it worked for a while, until we encountered a lot of 404 error like below

ERROR pid=14248 tid=MainThread file=base_modinput.py:log_error:309 | Error getting callRecord data: 404 Client Error: Not Found for url: https://graph.microsoft.com/v1.0/communications/callRecords/<call ID>?$expand=sessions($expand=segments)

 

I found out that the callID was removed from Teams CDR for some reason,  therefore when Splunk tried to download the CDR, it returned error 404, which is understandable.

However Teams Add-on will not remove the Call ID from webhook directory for this scenario. The call ID will remain there forever and Splunk will keep on trying again and again to download the CDR and failed. This results in a huge amount of call IDs that never get cleaned up and massive number of error messages in the log.

Further more, i found out that if there were too many call ID files exist in the wehbook directory (~60K), the Add-on will encountered error "401 Unauthorized to download the CDR" and stopped working soon afterward. After restarting Splunk, the Add-on worked again and then stopped the  moment it hit 401 error again. I manually created a script to manage the load of webhook folder, so this is OK for now but it would be preferable that the Add-on has load management feature by itself.

Hopefully the author of this Add-on will add this error handling soon, but meanwhile if anyone knows how to get around this 404 issue please kindly share.

Thanks a lot!

Labels (2)
0 Karma

tkomatsubara_sp
Splunk Employee
Splunk Employee

Hello,

 

The reason for that is the call ID is no longer available at Azure side, but MS Teams addon tries to get information with it.

Currently there is no way except to delete the local kv store lookup data.

 

Please try the below command and see if there is any improvements.

 

splunk clean kvstore -app TA_MS_Teams -collection TA_MS_Teams_checkpointer

 

0 Karma

Cbr1sg
Path Finder

I dont think this has anything to do with kvstore.

problem is clear: the add-on doesn't handle 404 error properly

Flow in normal situation:

check webhook folder for call ID --> download call ID --> delete call ID from webhook folder --> proceed with next ID

Flow in 404 situation:

check webhook folder for call ID --> download call ID (but failed) --> raise error.  And it stops there, the call ID with 404 error is not cleaned up from webhook folder.

So i think the author of the app just needs to improve the error handling to clean up "404" call ID from the webhook folder, problem will be solved

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!