How can I fix Issues fetching Exchange Online message tracking logs (HTTP 401 Client Error) since 1.4.2023?

[rvaglid]

Since the first of April we started receiving HTTP 401 Client Error in modular input logs from Splunk Add-on for Microsoft Office 365 Reporting Web Service (TA-MS_O365_Reporting version 2.0.1).
We tried both OAuth authentication and basic authentication, but we still receive the same error.

I was able to replicate the same issue in another Splunk environment against another M365 tenant.

We also configured the addon Splunk Add-on for Microsoft Office 365 (splunk_ta_o365 version 4.2.1) to fetch these logs, but we still receive the HTTP 401.

We are pretty confident that the app registrations and permissions are set up correctly.

Both apps connects to the API endpoint https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace - do anyone know of any changes made to this endpoint from Microsoft?

Cheers,

Rolf

[grokdesigns]

@Wiessiet, ETA I got is today (4/26) at 10am PST.

[grokdesigns]

@Wiessiet Did they happen to provide an internal reference number or anything? Experiencing the same issue and my support rep hasn’t been able to locate anything internally. Thanks!

[Wiessiet]

They did, the only direct reference they provided was a Jira case: ADDON-61818

[AhmadGul23]

Hi @Wiessiet ,

Did you get any update from Splunk on this issue yet? I've not been able to open a case yet so wanted to ask if anyone who had a case already opened received any update. 

I've observed another strange behaviour of the logs since that even though I receive the 401 client error and no successful ingest messages in the internal logs somehow I still receive the message trace logs for the time window that the API call is being made for like 5:30-6:30 and since the API call is being made repeatedly for the same duration I am getting duplicate logs for that time. So I've temp disabled the input. 

[MightyJ]

Same, raising a support ticket with Splunk atm.

Funny thing is the continuous message trace is still running even though it's constantly throwing 401 errors, but only 2000 events at a time and intermittent using all default settings in the input.

Using Azure Enterprise App with ID & Secrets

[Wiessiet]

My messaging team said that MS got back to us and the issue is the deprecation of Basic Auth in our tenant. We didn't get any heads up that I was aware of, so it took me by surprise. Unfortunately I've since installed the Splunk Add-on for Microsoft Office 365 (https://splunkbase.splunk.com/app/4055) and it doesn't work. Using an Azure application with OAuth I still get a 401 error. I've tested the exact same credentials with Postman and they work fine and message trace logs download, so there's something else going on. I have a little more troubleshooting to do and then I'll open a ticket with Splunk as well...

[madcitygeek]

Any update from this? I've got the exact same issue with the exact same troubleshooting with Oauth credentials. Postman using the Bearer token from the add-on (added to debug logs to test) and it works... most of the time. It seems resubmitting if I get the "Login to Outlook" page will generate the report.

I notice that 4.2.1 of the add-on fixes an issue for "401 authorization errors for Management Activity inputs." I'm wondering if there isn't a similar fix needed for 401s from message trace.

[amyers16]

I even tried setting up a totally new instance with new app registration and still getting the same error. Not sure that is overly helpful.

[dbot2001]

That is very helpful, saves us all the time in trying that as a workaround.

Thank you!

[LukaszZeszko]

After receiving same error, i have switched from user account to to app account and its working again. Check if app has proper permissions according to:

 https://www.michev.info/blog/post/4067/modern-authentication-oauth-support-for-the-reporting-web-ser...

Especially: "Remember that if you are running in the delegate permissions model, the user will need to have an appropriate admin role assigned as well. The supported roles for the Reporting Web Service are Global Reader and Security Reader."

and

"ReportingWebService.Read.All for application ones, can only be found under the Office 365 Exchange Online resource. As Microsoft has since hidden the relevant entry under the Request API permissions pane, you will have to follow the instructions from this article to get to it."

After switching to OAuth it went form 401 to 403, and after setting all roles and permissions right from 403 to working.

All using following splunk app :

Splunk Add-on for Microsoft Office 365

splunk_ta_o365

4.2.1

With input Message Trace.

[Wiessiet]

@LukaszZeszko  - are you currently consuming message trace logs successfully? I implemented this add-on and sorted out my permissions issues and I still get the same 401 error that I got with basic auth:

Basic auth:

2023-04-10 08:53:50,836 DEBUG pid=2504 tid=MainThread file=connectionpool.py:_make_request:437 | https://reports.office365.com:443 "GET /ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%20eq%20datetime'2023-03-31T15:51:52.522026Z'%20and%20EndDate%20eq%20datetime'2023-03-31T16:51:52.522026Z' HTTP/1.1" 401 1293

2023-04-10 08:53:50,838 ERROR pid=2504 tid=MainThread file=base_modinput.py:log_error:309 | HTTP Request error: 401 Client Error:  for url: https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%20eq%20datetime'2023-03-31T15:51:52.522026Z'%20and%20EndDate%20eq%20datetime'2023-03-31T16:51:52.522026Z'

2023-04-10 08:53:50,839 ERROR pid=2504 tid=MainThread file=base_modinput.py:log_error:309 | Get error when collecting events.

OAuth:

2023-04-10 16:36:55,126 level=DEBUG pid=22685 tid=MainThread logger=splunk_ta_o365.modinputs.message_trace pos=__init__.py:_process_messages:298 | datainput=b'XXXX_message_trace' start_time=1681159011 | message="nextLink URL (@odata.nextLink): https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%20eq%20datetime'2023-04-09T11%3A00%3A00Z'%20and%20EndDate%20eq%20datetime'2023-04-09T12%3A00%3A00Z'&$skiptoken=1999"

2023-04-10 16:36:55,424 level=ERROR pid=22685 tid=MainThread logger=splunk_ta_o365.modinputs.message_trace pos=__init__.py:_get_messages:253 | datainput=b'XXXX_message_trace' start_time=1681159011 | message="HTTP Request error: 401 Client Error:  for url: https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%20eq%20datetime'2023-04-09T11%3A00%3A00Z'%20and%20EndDate%20eq%20datetime'2023-04-09T12%3A00%3A00Z'&$skiptoken=1999" stack_info=True

[Wiessiet]

@LukaszZeszko- would you be able to share the permissions you set up to avoid the 403 error? I've gotten stuck there. I have "Office 365 Exchange Online --> ReportingWebService.Read.All" permissions for my application. It seems I need more permissions for this to work but it's unclear to me from the documentation what permissions that should be. Thanks in advance!

[raz_gp]

re @Wiessiet

I also looked at that for a while and found these 2 articles which helped me: 

https://www.michev.info/blog/post/3180/exchange-api-permissions-missing
https://www.michev.info/blog/post/4067/modern-authentication-oauth-support-for-the-reporting-web-ser...
 

In short, these are the steps:

To get there, App Registrations -> Select our app -> API permissions
Then Add a permission -> go to APIs my organization uses -> paste: Office 365 Exchange Online -> Click the Office 365 Exchange Online API section -> then Application permissions
Paste the ReportingWebService.Read.All permissions and tick it + approve afterwards.

It took me a while as well. 

+ the Global reader permission addition from Roles & admins -> Global Reader -> add assignments  -> paste the app name and  grant your app the role rights... 

https://learn.microsoft.com/en-us/entra/identity/role-based-access-control/manage-roles-portal

[Wiessiet]

That's great information. I had reviewed that update to the app and to using OAuth but never got around to implementing. I'm going to test that in my non-production environment today and I'll report back.

[Martijn]

We do have exactly the same issue. I hope it will get resolved soon.

[Wiessiet]

I was glad to find your post - we're having the exact same issue. My first instance of receiving a 401 error for that integration was on 2023-03-31 13:22:20 (Eastern). My O365 administration team pointed me to this posting from Microsoft (looks like you'll need a certain level of access in your tenant to view this) https://admin.microsoft.com/#/servicehealth/:/alerts/EX537209.

It would seem that this is a Microsoft issue but any kind of information is hard to come by. I hope that's of some use. We're working on reporting it to MS but we're basically in a holding pattern until this works again.