All Apps and Add-ons

Microsoft Office 365 Reporting Add-on for Splunk not pulling data - exiting with 500 Internal Server Error

Path Finder

Installed and configured Microsoft Office 365 Reporting Add-on for Splunk but it doesn't seem to be pulling any data. Here's the error we see in the ta_ms_o365_reporting_ms_o365_message_trace.log file:

2019-10-30 16:21:16,836 INFO pid=40891 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-10-30 16:21:17,674 INFO pid=40891 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-10-30 16:21:19,141 INFO pid=40891 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-10-30 16:21:20,569 INFO pid=40891 tid=MainThread file=splunk_rest_client.py:_request_handler:100 | Use HTTP connection pooling
2019-10-30 16:21:20,570 DEBUG pid=40891 tid=MainThread file=binding.py:get:664 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/config/TA_MS_O365_Reporting_checkpointer (body: {})
2019-10-30 16:21:20,571 INFO pid=40891 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-10-30 16:21:20,576 DEBUG pid=40891 tid=MainThread file=connectionpool.py:_make_request:387 | "GET /servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/config/TA_MS_O365_Reporting_checkpointer HTTP/1.1" 200 5516
2019-10-30 16:21:20,577 DEBUG pid=40891 tid=MainThread file=binding.py:new_f:71 | Operation took 0:00:00.006580
2019-10-30 16:21:20,577 DEBUG pid=40891 tid=MainThread file=binding.py:get:664 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/config/ (body: {'count': -1, 'search': 'TA_MS_O365_Reporting_checkpointer', 'offset': 0})
2019-10-30 16:21:20,580 DEBUG pid=40891 tid=MainThread file=connectionpool.py:_make_request:387 | "GET /servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/config/?count=-1&search=TA_MS_O365_Reporting_checkpointer&offset=0 HTTP/1.1" 200 7417
2019-10-30 16:21:20,580 DEBUG pid=40891 tid=MainThread file=binding.py:new_f:71 | Operation took 0:00:00.003192
2019-10-30 16:21:20,583 DEBUG pid=40891 tid=MainThread file=binding.py:get:664 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/data/TA_MS_O365_Reporting_checkpointer/o365_message_trace_obj_checkpoint (body: {})
2019-10-30 16:21:20,585 DEBUG pid=40891 tid=MainThread file=connectionpool.py:_make_request:387 | "GET /servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/data/TA_MS_O365_Reporting_checkpointer/o365_message_trace_obj_checkpoint HTTP/1.1" 404 140
2019-10-30 16:21:20,587 DEBUG pid=40891 tid=MainThread file=base_modinput.py:log_debug:286 | Start date: 2019-09-10 00:00:00, End date: 2019-09-10 01:00:00
2019-10-30 16:21:20,587 DEBUG pid=40891 tid=MainThread file=base_modinput.py:log_debug:286 | Endpoint URL: https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate eq datetime'2019-09-10T00:00:00Z' and EndDate eq datetime'2019-09-10T01:00:00Z'
2019-10-30 16:21:20,587 INFO pid=40891 tid=MainThread file=setup_util.py:log_info:114 | Proxy is not enabled!
2019-10-30 16:21:20,596 DEBUG pid=40891 tid=MainThread file=connectionpool.py:_new_conn:809 | Starting new HTTPS connection (1): reports.office365.com
2019-10-30 16:21:20,976 DEBUG pid=40891 tid=MainThread file=connectionpool.py:_make_request:400 | https://reports.office365.com:443 "GET /ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%20eq%20datetime'2019-09-10T00:00:00Z'%20and%20EndDate%20eq%20datetime'2019-09-10T01:00:00Z' HTTP/1.1" 500 113
2019-10-30 16:21:20,979 ERROR pid=40891 tid=MainThread file=base_modinput.py:log_error:307 | HTTP Request error: 500 Server Error: Internal Server Error for url: https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%20eq%20datetime'2019-09-10T00:00:00Z'%20and%20EndDate%20eq%20datetime'2019-09-10T01:00:00Z'
1 Solution

Path Finder

We opened a case with Microsoft on 10/31 and the case was resolved by 11/02 after which we were no longer getting 500 Internal Server Error and the Add-On was pulling data once again. Here's the Preliminary Post Incident Review Report from Microsoft related to this incident.

View solution in original post

Path Finder

We opened a case with Microsoft on 10/31 and the case was resolved by 11/02 after which we were no longer getting 500 Internal Server Error and the Add-On was pulling data once again. Here's the Preliminary Post Incident Review Report from Microsoft related to this incident.

View solution in original post

Explorer

Short Update to the Server Error. The error even appears when browsing manual to reports.office365.com.

 

add a \ before the $filter and the error is gone. e.g:

doesnt work: https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate eq datetime'2020-05-28T21:50:04.772888Z' and EndDate eq datetime'2020-05-28T22:50:04.772888Z'

 

works: https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?\$filter=StartDate eq datetime'2020-05-28T21:50:04.772888Z' and EndDate eq datetime'2020-05-28T22:50:04.772888Z'

 

Path Finder

Adding the backslash into the input_module_ms_o365_message_trace.py at lines 156 & 225 solved this for me (at least today). BTW, using v1.2.1 of the add-on.

Thanks for this solution @poisar !!

Explorer

Mine Just started working. Not sure what was changed I am reaching out to our MS team to see if they changed anything

0 Karma

Explorer

The API starting working for us again. MS clearly responsible.

New Member

Same thing here.

Still have not heard back from MS though.

0 Karma

Explorer

Seems that this reporting API is totally best effort SLA. Not a nice solution to rely on.

0 Karma

Explorer

We're getting 500 errors too. When I test with postman to the api without searching I can authenticate ok. Looks like someothing has changed?

https://reports.office365.com/ecp/reportingwebservice/reporting.svc/

I have a support issue open but not much progress there. Would be interesting to know how many people have this issue right now.

0 Karma

Path Finder

@tommusgrave Let us know if you hear back from Microsoft Support. Also upvote the question to keep a count of people affected by this issue

0 Karma

New Member

I am having the same exact issue. We had the add on working properly for at least 6 months but it started returning an error starting a few days ago.

I have opened a ticket with our Microsoft Support team to see if they can shed some light on this.

Will post my results here when I get more information.

0 Karma

Explorer

Same problem here: i have little to no information on the API changes on o365 reporting service

0 Karma

Path Finder

@raugugliaro_ao Let us know if you hear back from Microsoft Support. Also upvote the question to keep a count of people affected by this issue

0 Karma

Explorer

I stumbled on this. It looks like Microsoft has made some changes to the API
https://techcommunity.microsoft.com/t5/Office-365-Blog/Announcing-the-General-Availability-of-Micros...

0 Karma

Path Finder

@jonesy1111 The document referenced above says that the MessageTrace method will continue to work as expected and is not impacted by this deprecation

https://docs.microsoft.com/en-us/previous-versions/office/developer/o365-enterprise-developers/jj984...

0 Karma

Explorer

Oh.. neat, I missed that part. I wonder if the issue is the url format of the query. I am by no means a SME. Just trying to find a solution.

0 Karma

Path Finder

@jonesy1111 Not a problem. Don't think it's an issue with the url format since if you directly hit the API endpoint, it shows the error. Please upvote the question to keep a count of people affected by this issue

0 Karma

Splunk Employee
Splunk Employee

A 500 error is going to be on the server side - in other words on the API side. The API web service uses basic auth, so it's pretty easy to test with just a browser. Navigate to https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace and log in with an account that has permission. If you get an error there, the add-on will get the same error.

0 Karma

Splunk Employee
Splunk Employee

Several customers have reported the API is working again on the MSFT side. Your add-on should start catching up since it saved the check point.

If this URL works for you in a browser or Postman, the add-on should be working too -> https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace

0 Karma

Path Finder

@jconger I did that and got the following error:

<m:error xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata">
<m:code/>
<m:message xml:lang="en-US">An error occurred while processing this request.</m:message>
</m:error>
0 Karma

Path Finder

@jconger @lnetto_splunk

0 Karma