Microsoft message trace logs pull issue - 401 Client Error?

[sumeet1503]

We are ingesting Microsoft tracelogs on Splunk using a Splunk Addon - "Splunk Addon for Microsoft Office 365 Reporting Web Service" .
Addon link -https://splunkbase.splunk.com/app/3720

The Addon leverages the credentials from an Application registered on Azure AD and tries to connect to below URL to pull the trace logs from Microsoft reporting webservices. It polls the API every 5 mins. -
URL -: https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%2...

Since Apr,6th we observed a sigificant reduction in the log volume on Splunk and to give some stats the app used to pull 50-60 lakhs of events per day which has come down to 1-2 lakh events per day.

On further investigation we found some error(401 Client Error) in the logs on Splunk App and it looks like its failing to connect to the URL intermittently leading to drop in the events . PFB error details –

requests.exceptions.HTTPError: 401 Client Error: for url: https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%2...

We have also tried to configure the messagetrace inputs on another Addon - "Splunk Add-on for Microsoft Office 365" and have seen the same errors (401 Client Error) .

We have also validated the credentials (Tenant id , Client id and Secret Key) and all looks good .

It looks like this issue was identified by other people as well and we can see this was added as known issue on Splunk doc- Splunk Add-on for Microsoft Office 365 .

Has anyone identified a fix to it or know a workaround. ? Any help would be appreciated. 

[Junie]

We are using the Splunk Add-on for Microsoft Office 365 splunk_ta_o365 version 4.2.0 and having the exact same issue as originally posted by sumeet1503 (ours also started on Apr 6th).

[sdesruelles]

Hi @sumeet1503 ,

There is another topic on this subject https://community.splunk.com/t5/All-Apps-and-Add-ons/Issues-fetching-Exchange-Online-message-trackin...

It seems Microsoft changed something and Splunk is working on an update for the app.

Maybe the old app is still working https://splunkbase.splunk.com/app/3720 haven't tried it yet.

Best regards,

S.

[sumeet1503]

We have opened ticket with both Microsoft and Splunk and getting this checked .

Microsoft exchange team is currently looking into it but they havent find anything on this yet and says this problem is on splunk addon.  However they are still looking into it .

Splunk support replicated the same issue using automated postman call and suspect some issue on the MS reporting webservice API end .

They say they are already in touch with MS to get this identified and fix .

So far it looks like its kinda going in circles . We tried creating different App registration on Azure AD to see if that works but found no luck.

I’ll update here if we get a fix to it.

Happy Splunking…

[sumeet1503]

We had a call with splunk support just now and they updated the engineering team is working closely with Microsoft team to find the issue and fix . 

The latest they have from Microsoft is that MS is working on some patch that will be rolled out in a day or two. More updates to come but this should hopefully fix it.

However i did observe  increase in the log volume today (26/04/23) compared to the total count since Apr,6th and I can see the count is growing todaywhich is a good sign , so looks like something is in progress. 😊

Will keep everyone posted on updates.

Regards,

Sumeet

[sumeet1503]

Microsoft identified the issue and has patched it day before yesterday .
The message trace logs are back to normal volume now and we arent seeing 401 client error anymore.

We have validated  it in our environment .

Apparently the root cause identified from MS was like they did some changes on one of their system (ECS) on 4th apr that caused some code conflict and sometimes the calls made to microsoft webservice reporting API were getting redirected to Outlook login page which was throwing the 401 error.

Hope everyone else has got this working too.😊

Thanks,

Sumeet