Microsoft Office 365 Reporting Add-on for Splunk not pulling data - exiting with 500 Internal Server Error

[nmadhok]

Installed and configured Microsoft Office 365 Reporting Add-on for Splunk but it doesn't seem to be pulling any data. Here's the error we see in the ta_ms_o365_reporting_ms_o365_message_trace.log file:

2019-10-30 16:21:16,836 INFO pid=40891 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-10-30 16:21:17,674 INFO pid=40891 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-10-30 16:21:19,141 INFO pid=40891 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-10-30 16:21:20,569 INFO pid=40891 tid=MainThread file=splunk_rest_client.py:_request_handler:100 | Use HTTP connection pooling
2019-10-30 16:21:20,570 DEBUG pid=40891 tid=MainThread file=binding.py:get:664 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/config/TA_MS_O365_Reporting_checkpointer (body: {})
2019-10-30 16:21:20,571 INFO pid=40891 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-10-30 16:21:20,576 DEBUG pid=40891 tid=MainThread file=connectionpool.py:_make_request:387 | "GET /servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/config/TA_MS_O365_Reporting_checkpointer HTTP/1.1" 200 5516
2019-10-30 16:21:20,577 DEBUG pid=40891 tid=MainThread file=binding.py:new_f:71 | Operation took 0:00:00.006580
2019-10-30 16:21:20,577 DEBUG pid=40891 tid=MainThread file=binding.py:get:664 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/config/ (body: {'count': -1, 'search': 'TA_MS_O365_Reporting_checkpointer', 'offset': 0})
2019-10-30 16:21:20,580 DEBUG pid=40891 tid=MainThread file=connectionpool.py:_make_request:387 | "GET /servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/config/?count=-1&search=TA_MS_O365_Reporting_checkpointer&offset=0 HTTP/1.1" 200 7417
2019-10-30 16:21:20,580 DEBUG pid=40891 tid=MainThread file=binding.py:new_f:71 | Operation took 0:00:00.003192
2019-10-30 16:21:20,583 DEBUG pid=40891 tid=MainThread file=binding.py:get:664 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/data/TA_MS_O365_Reporting_checkpointer/o365_message_trace_obj_checkpoint (body: {})
2019-10-30 16:21:20,585 DEBUG pid=40891 tid=MainThread file=connectionpool.py:_make_request:387 | "GET /servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/data/TA_MS_O365_Reporting_checkpointer/o365_message_trace_obj_checkpoint HTTP/1.1" 404 140
2019-10-30 16:21:20,587 DEBUG pid=40891 tid=MainThread file=base_modinput.py:log_debug:286 | Start date: 2019-09-10 00:00:00, End date: 2019-09-10 01:00:00
2019-10-30 16:21:20,587 DEBUG pid=40891 tid=MainThread file=base_modinput.py:log_debug:286 | Endpoint URL: https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate eq datetime'2019-09-10T00:00:00Z' and EndDate eq datetime'2019-09-10T01:00:00Z'
2019-10-30 16:21:20,587 INFO pid=40891 tid=MainThread file=setup_util.py:log_info:114 | Proxy is not enabled!
2019-10-30 16:21:20,596 DEBUG pid=40891 tid=MainThread file=connectionpool.py:_new_conn:809 | Starting new HTTPS connection (1): reports.office365.com
2019-10-30 16:21:20,976 DEBUG pid=40891 tid=MainThread file=connectionpool.py:_make_request:400 | https://reports.office365.com:443 "GET /ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%20eq%20datetime'2019-09-10T00:00:00Z'%20and%20EndDate%20eq%20datetime'2019-09-10T01:00:00Z' HTTP/1.1" 500 113
2019-10-30 16:21:20,979 ERROR pid=40891 tid=MainThread file=base_modinput.py:log_error:307 | HTTP Request error: 500 Server Error: Internal Server Error for url: https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%20eq%20datetime'2019-09-10T00:00:00Z'%20and%20EndDate%20eq%20datetime'2019-09-10T01:00:00Z'

[mik3y]

I can confirm this was affecting version 2.0.1 and 2.0.0 and appeared to break around 7/Sept/2022 for us.

Modifying the following two files resolved the issue

(grep "microsoft_trace_url =" /opt/splunk/etc/apps/TA-MS_O365_Reporting/bin/*.py)

OR

/opt/splunk/etc/apps/TA-MS_O365_Reporting/bin/ms_o365_message_trace_oauth.py

/opt/splunk/etc/apps/TA-MS_O365_Reporting/bin/ms_o365_message_trace.py

Changing the URL (to include a backslash) from

https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?$filter

to

https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?\$filter

Will lodge a splunk support ticket now.

Cheers

[mik3y]

I guess Splunk don't support Splunk or Splunk Works developed apps now.

[haraksin]

I opened a case with microsoft for this error - the issue for me was that the docs (both on splunkbase and in the Microsoft docs article referenced by Splunkbase) say that the Message Trace API can gather data up to 30 days prior. This is incorrect. The correct documentation can be found here, which shows that only a max of 10 days prior is allowed: https://docs.microsoft.com/en-us/powershell/module/exchange/get-messagetrace?redirectedfrom=MSDN&vie...

[nmadhok · Accepted solution]

We opened a case with Microsoft on 10/31 and the case was resolved by 11/02 after which we were no longer getting 500 Internal Server Error and the Add-On was pulling data once again. Here's the Preliminary Post Incident Review Report from Microsoft related to this incident.

[poisar]

Short Update to the Server Error. The error even appears when browsing manual to reports.office365.com.

add a \ before the $filter and the error is gone. e.g:

doesnt work: https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate eq datetime'2020-05-28T21:50:04.772888Z' and EndDate eq datetime'2020-05-28T22:50:04.772888Z'

works: https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?\$filter=StartDate eq datetime'2020-05-28T21:50:04.772888Z' and EndDate eq datetime'2020-05-28T22:50:04.772888Z'

[gordo32]

Adding the backslash into the input_module_ms_o365_message_trace.py at lines 156 & 225 solved this for me (at least today). BTW, using v1.2.1 of the add-on.

Thanks for this solution @poisar !!

[jonesy1111]

Mine Just started working. Not sure what was changed I am reaching out to our MS team to see if they changed anything

[tommusgrave]

The API starting working for us again. MS clearly responsible.

[raugugliaro_ao]

Same thing here.

Still have not heard back from MS though.

[tommusgrave]

Seems that this reporting API is totally best effort SLA. Not a nice solution to rely on.

[tommusgrave]

We're getting 500 errors too. When I test with postman to the api without searching I can authenticate ok. Looks like someothing has changed?

https://reports.office365.com/ecp/reportingwebservice/reporting.svc/

I have a support issue open but not much progress there. Would be interesting to know how many people have this issue right now.

[nmadhok]

@tommusgrave Let us know if you hear back from Microsoft Support. Also upvote the question to keep a count of people affected by this issue

[raugugliaro_ao]

I am having the same exact issue. We had the add on working properly for at least 6 months but it started returning an error starting a few days ago.

I have opened a ticket with our Microsoft Support team to see if they can shed some light on this.

Will post my results here when I get more information.

[jaivijay_rio]

Same problem here: i have little to no information on the API changes on o365 reporting service

[nmadhok]

@raugugliaro_ao Let us know if you hear back from Microsoft Support. Also upvote the question to keep a count of people affected by this issue

[jonesy1111]

I stumbled on this. It looks like Microsoft has made some changes to the API
https://techcommunity.microsoft.com/t5/Office-365-Blog/Announcing-the-General-Availability-of-Micros...

[nmadhok]

@jonesy1111 The document referenced above says that the MessageTrace method will continue to work as expected and is not impacted by this deprecation

https://docs.microsoft.com/en-us/previous-versions/office/developer/o365-enterprise-developers/jj984...

[jonesy1111]

Oh.. neat, I missed that part. I wonder if the issue is the url format of the query. I am by no means a SME. Just trying to find a solution.

[nmadhok]

@jonesy1111 Not a problem. Don't think it's an issue with the url format since if you directly hit the API endpoint, it shows the error. Please upvote the question to keep a count of people affected by this issue

[jconger]

A 500 error is going to be on the server side - in other words on the API side. The API web service uses basic auth, so it's pretty easy to test with just a browser. Navigate to https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace and log in with an account that has permission. If you get an error there, the add-on will get the same error.