topic Re: Issues fetching Exchange Online message tracking logs (HTTP 401 Client Error) since 1.4.2023 in All Apps and Add-ons

How can I fix Issues fetching Exchange Online message tracking logs (HTTP 401 Client Error) since 1.4.2023?

rvaglid — Wed, 26 Apr 2023 17:09:05 GMT

Since the first of April we started receiving HTTP 401 Client Error in modular input logs from Splunk Add-on for Microsoft Office 365 Reporting Web Service (TA-MS_O365_Reporting version 2.0.1).
We tried both OAuth authentication and basic authentication, but we still receive the same error.

I was able to replicate the same issue in another Splunk environment against another M365 tenant.

We also configured the addon Splunk Add-on for Microsoft Office 365 (splunk_ta_o365 version 4.2.1) to fetch these logs, but we still receive the HTTP 401.

We are pretty confident that the app registrations and permissions are set up correctly.

Both apps connects to the API endpoint https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace - do anyone know of any changes made to this endpoint from Microsoft?

Cheers,

Rolf

Re: Issues fetching Exchange Online message tracking logs (HTTP 401 Client Error) since 1.4.2023

Wiessiet — Thu, 06 Apr 2023 16:02:02 GMT

I was glad to find your post - we're having the exact same issue. My first instance of receiving a 401 error for that integration was on 2023-03-31 13:22:20 (Eastern). My O365 administration team pointed me to this posting from Microsoft (looks like you'll need a certain level of access in your tenant to view this) https://admin.microsoft.com/#/servicehealth/:/alerts/EX537209.

It would seem that this is a Microsoft issue but any kind of information is hard to come by. I hope that's of some use. We're working on reporting it to MS but we're basically in a holding pattern until this works again.

Re: Issues fetching Exchange Online message tracking logs (HTTP 401 Client Error) since 1.4.2023

Martijn — Thu, 06 Apr 2023 19:03:10 GMT

We do have exactly the same issue. I hope it will get resolved soon.

Re: Issues fetching Exchange Online message tracking logs (HTTP 401 Client Error) since 1.4.2023

LukaszZeszko — Fri, 07 Apr 2023 09:45:42 GMT

After receiving same error, i have switched from user account to to app account and its working again. Check if app has proper permissions according to:

https://www.michev.info/blog/post/4067/modern-authentication-oauth-support-for-the-reporting-web-ser...

Especially: "Remember that if you are running in the delegate permissions model, the user will need to have an appropriate admin role assigned as well. The supported roles for the Reporting Web Service are Global Reader and Security Reader."

and

"ReportingWebService.Read.All for application ones, can only be found under the Office 365 Exchange Online resource. As Microsoft has since hidden the relevant entry under the Request API permissions pane, you will have to follow the instructions from this article to get to it."

After switching to OAuth it went form 401 to 403, and after setting all roles and permissions right from 403 to working.

All using following splunk app :

Splunk Add-on for Microsoft Office 365

splunk_ta_o365

4.2.1

With input Message Trace.

Re: Issues fetching Exchange Online message tracking logs (HTTP 401 Client Error) since 1.4.2023

Wiessiet — Fri, 07 Apr 2023 12:57:15 GMT

That's great information. I had reviewed that update to the app and to using OAuth but never got around to implementing. I'm going to test that in my non-production environment today and I'll report back.

Re: Issues fetching Exchange Online message tracking logs (HTTP 401 Client Error) since 1.4.2023

Wiessiet — Fri, 07 Apr 2023 19:55:15 GMT

@LukaszZeszko- would you be able to share the permissions you set up to avoid the 403 error? I've gotten stuck there. I have "Office 365 Exchange Online --> ReportingWebService.Read.All" permissions for my application. It seems I need more permissions for this to work but it's unclear to me from the documentation what permissions that should be. Thanks in advance!

Re: Issues fetching Exchange Online message tracking logs (HTTP 401 Client Error) since 1.4.2023

Wiessiet — Tue, 11 Apr 2023 14:18:16 GMT

@LukaszZeszko - are you currently consuming message trace logs successfully? I implemented this add-on and sorted out my permissions issues and I still get the same 401 error that I got with basic auth:

Basic auth:

2023-04-10 08:53:50,836 DEBUG pid=2504 tid=MainThread file=connectionpool.py:_make_request:437 | https://reports.office365.com:443 "GET /ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%20eq%20datetime'2023-03-31T15:51:52.522026Z'%20and%20EndDate%20eq%20datetime'2023-03-31T16:51:52.522026Z' HTTP/1.1" 401 1293 2023-04-10 08:53:50,838 ERROR pid=2504 tid=MainThread file=base_modinput.py:log_error:309 | HTTP Request error: 401 Client Error: for url: https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%20eq%20datetime'2023-03-31T15:51:52.522026Z'%20and%20EndDate%20eq%20datetime'2023-03-31T16:51:52.522026Z' 2023-04-10 08:53:50,839 ERROR pid=2504 tid=MainThread file=base_modinput.py:log_error:309 | Get error when collecting events.

OAuth:

2023-04-10 16:36:55,126 level=DEBUG pid=22685 tid=MainThread logger=splunk_ta_o365.modinputs.message_trace pos=__init__.py:_process_messages:298 | datainput=b'XXXX_message_trace' start_time=1681159011 | message="nextLink URL (@odata.nextLink): https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%20eq%20datetime'2023-04-09T11%3A00%3A00Z'%20and%20EndDate%20eq%20datetime'2023-04-09T12%3A00%3A00Z'&$skiptoken=1999" 2023-04-10 16:36:55,424 level=ERROR pid=22685 tid=MainThread logger=splunk_ta_o365.modinputs.message_trace pos=__init__.py:_get_messages:253 | datainput=b'XXXX_message_trace' start_time=1681159011 | message="HTTP Request error: 401 Client Error: for url: https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%20eq%20datetime'2023-04-09T11%3A00%3A00Z'%20and%20EndDate%20eq%20datetime'2023-04-09T12%3A00%3A00Z'&$skiptoken=1999" stack_info=True

Re: Issues fetching Exchange Online message tracking logs (HTTP 401 Client Error) since 1.4.2023

amyers16 — Wed, 12 Apr 2023 17:52:10 GMT

I am facing the same issue, but according to the link @Wiessiet posted, it was resolved on April 6th and they are not updating the case information any longer.

Re: Issues fetching Exchange Online message tracking logs (HTTP 401 Client Error) since 1.4.2023

Wiessiet — Wed, 12 Apr 2023 18:15:46 GMT

I saw the same thing yesterday when I went to reference the status. My organization has a ticket open with Microsoft and our O365 team forwarded me the update from a Microsoft engineer that they could replicate the issue. If our ticket bears fruit and I have a worthwhile update for everyone I'll post it here.

Re: Issues fetching Exchange Online message tracking logs (HTTP 401 Client Error) since 1.4.2023

scannon4 — Fri, 14 Apr 2023 13:28:41 GMT

Anyone have an update on this? We have this issue when using Message Trace inside the Add on for Microsoft Office 365. But the older app Splunk Add On for Microsoft Office 365 Reporting Web Service works.

Re: Issues fetching Exchange Online message tracking logs (HTTP 401 Client Error) since 1.4.2023

bbour53 — Fri, 14 Apr 2023 19:53:17 GMT

Throwing my 2-cents in here. We've been using the OAuth route since the beginning of the year without issue. We didn't start seeing the 401 error until 4/7.

After the errors started on 4/7, it looked like the error was intermittent and ingestion continued until 4/10, at which point the errors became very persistent.

Disabling the input and re-enabling seemed to have temporarily brought some belief but the issue persists. Sometimes, based on the log, you can see the skiptoken successfully incrementing until it randomly hits the 401. Debug logs show successful retrieval of the access token and Azure AD logs confirm that the app isn't getting any auth failures.

Ideally the consistency of the Microsoft endpoint improves, but maybe the Splunk Add-On for Microsoft Office 365 needs a better method to catch this error and retry instead of starting the collection again at the first message after every failure.

Re: Issues fetching Exchange Online message tracking logs (HTTP 401 Client Error) since 1.4.2023

rvaglid — Fri, 14 Apr 2023 22:18:57 GMT

Thank you all for your input on this issue.

We managed to get it working again after a great deal of trial and error.
We ended up creating a brand new service principal, and applying the same permissions again. We had to manually alter the manifest to be able to select "ReportingWebService.Read.All".

Our working theory is that the service principal we have been using for years might be "outdated" in a way, as it started working instantly with a brand new service principal.
We do not have any information from MS on this theory, as we were able to resolve the issue just before submitting a case to MS.

Cheers,
Rolf

Re: Issues fetching Exchange Online message tracking logs (HTTP 401 Client Error) since 1.4.2023

Wiessiet — Tue, 18 Apr 2023 15:15:09 GMT

My messaging team said that MS got back to us and the issue is the deprecation of Basic Auth in our tenant. We didn't get any heads up that I was aware of, so it took me by surprise. Unfortunately I've since installed the Splunk Add-on for Microsoft Office 365 (https://splunkbase.splunk.com/app/4055) and it doesn't work. Using an Azure application with OAuth I still get a 401 error. I've tested the exact same credentials with Postman and they work fine and message trace logs download, so there's something else going on. I have a little more troubleshooting to do and then I'll open a ticket with Splunk as well...

Re: Issues fetching Exchange Online message tracking logs (HTTP 401 Client Error) since 1.4.2023

amyers16 — Tue, 18 Apr 2023 18:06:35 GMT

I even tried setting up a totally new instance with new app registration and still getting the same error. Not sure that is overly helpful.

Re: Issues fetching Exchange Online message tracking logs (HTTP 401 Client Error) since 1.4.2023

MightyJ — Tue, 18 Apr 2023 19:17:04 GMT

Same, raising a support ticket with Splunk atm.

Funny thing is the continuous message trace is still running even though it's constantly throwing 401 errors, but only 2000 events at a time and intermittent using all default settings in the input.

Using Azure Enterprise App with ID & Secrets

Re: Issues fetching Exchange Online message tracking logs (HTTP 401 Client Error) since 1.4.2023

Wiessiet — Tue, 18 Apr 2023 19:30:12 GMT

Update from me - I just met with my Splunk support team for something else and made a mention about this. Apparently Splunk is aware and has an internal Jira open. It would appear MS changed something on their side that broke Splunk's TA. I'm going to open a case with Splunk anyway to help with visibility, but this might be a case of having to wait for the two companies to sort it out and fix it >.<

Re: Issues fetching Exchange Online message tracking logs (HTTP 401 Client Error) since 1.4.2023

grokdesigns — Wed, 19 Apr 2023 05:56:36 GMT

@Wiessiet Did they happen to provide an internal reference number or anything? Experiencing the same issue and my support rep hasn’t been able to locate anything internally. Thanks!

Re: Issues fetching Exchange Online message tracking logs (HTTP 401 Client Error) since 1.4.2023

Wiessiet — Wed, 19 Apr 2023 12:53:43 GMT

They did, the only direct reference they provided was a Jira case: ADDON-61818

Re: Issues fetching Exchange Online message tracking logs (HTTP 401 Client Error) since 1.4.2023

dbot2001 — Wed, 19 Apr 2023 15:18:33 GMT

That is very helpful, saves us all the time in trying that as a workaround.

Thank you!

Re: Issues fetching Exchange Online message tracking logs (HTTP 401 Client Error) since 1.4.2023

Wiessiet — Wed, 19 Apr 2023 18:48:04 GMT

By way of a further update - I logged a case with Splunk support today and got the following response:

Thank you for submitting the case. We are aware of this issue and I want to let you know that we have received many cases of the same issue from other customers as well.

We have reproduced & encountered the same error and suspect an issue with the API, not with the add-on. I would request that you allow us some time to validate the issue from the Microsoft Azure end to know the API behaviour.

In order to expedite the case, we have escalated the issue to our internal team, and the add-on engineering team has started the conversation with Microsoft about the 401 client error (message trace failure). Rest assured that I will keep you informed of any further updates on this matter.

I have also associated the internal Jira ticket for this issue with your support case so now even your account owner can check the status for any update from our internal add-on engineering team regarding this.

If you have any other questions, kindly let me know.