All Apps and Add-ons

Microsoft Office 365 Reporting Add-on for Splunk not pulling data - exiting with 500 Internal Server Error

nmadhok
Path Finder

Installed and configured Microsoft Office 365 Reporting Add-on for Splunk but it doesn't seem to be pulling any data. Here's the error we see in the ta_ms_o365_reporting_ms_o365_message_trace.log file:

2019-10-30 16:21:16,836 INFO pid=40891 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-10-30 16:21:17,674 INFO pid=40891 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-10-30 16:21:19,141 INFO pid=40891 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-10-30 16:21:20,569 INFO pid=40891 tid=MainThread file=splunk_rest_client.py:_request_handler:100 | Use HTTP connection pooling
2019-10-30 16:21:20,570 DEBUG pid=40891 tid=MainThread file=binding.py:get:664 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/config/TA_MS_O365_Reporting_checkpointer (body: {})
2019-10-30 16:21:20,571 INFO pid=40891 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-10-30 16:21:20,576 DEBUG pid=40891 tid=MainThread file=connectionpool.py:_make_request:387 | "GET /servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/config/TA_MS_O365_Reporting_checkpointer HTTP/1.1" 200 5516
2019-10-30 16:21:20,577 DEBUG pid=40891 tid=MainThread file=binding.py:new_f:71 | Operation took 0:00:00.006580
2019-10-30 16:21:20,577 DEBUG pid=40891 tid=MainThread file=binding.py:get:664 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/config/ (body: {'count': -1, 'search': 'TA_MS_O365_Reporting_checkpointer', 'offset': 0})
2019-10-30 16:21:20,580 DEBUG pid=40891 tid=MainThread file=connectionpool.py:_make_request:387 | "GET /servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/config/?count=-1&search=TA_MS_O365_Reporting_checkpointer&offset=0 HTTP/1.1" 200 7417
2019-10-30 16:21:20,580 DEBUG pid=40891 tid=MainThread file=binding.py:new_f:71 | Operation took 0:00:00.003192
2019-10-30 16:21:20,583 DEBUG pid=40891 tid=MainThread file=binding.py:get:664 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/data/TA_MS_O365_Reporting_checkpointer/o365_message_trace_obj_checkpoint (body: {})
2019-10-30 16:21:20,585 DEBUG pid=40891 tid=MainThread file=connectionpool.py:_make_request:387 | "GET /servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/data/TA_MS_O365_Reporting_checkpointer/o365_message_trace_obj_checkpoint HTTP/1.1" 404 140
2019-10-30 16:21:20,587 DEBUG pid=40891 tid=MainThread file=base_modinput.py:log_debug:286 | Start date: 2019-09-10 00:00:00, End date: 2019-09-10 01:00:00
2019-10-30 16:21:20,587 DEBUG pid=40891 tid=MainThread file=base_modinput.py:log_debug:286 | Endpoint URL: https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate eq datetime'2019-09-10T00:00:00Z' and EndDate eq datetime'2019-09-10T01:00:00Z'
2019-10-30 16:21:20,587 INFO pid=40891 tid=MainThread file=setup_util.py:log_info:114 | Proxy is not enabled!
2019-10-30 16:21:20,596 DEBUG pid=40891 tid=MainThread file=connectionpool.py:_new_conn:809 | Starting new HTTPS connection (1): reports.office365.com
2019-10-30 16:21:20,976 DEBUG pid=40891 tid=MainThread file=connectionpool.py:_make_request:400 | https://reports.office365.com:443 "GET /ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%20eq%20datetime'2019-09-10T00:00:00Z'%20and%20EndDate%20eq%20datetime'2019-09-10T01:00:00Z' HTTP/1.1" 500 113
2019-10-30 16:21:20,979 ERROR pid=40891 tid=MainThread file=base_modinput.py:log_error:307 | HTTP Request error: 500 Server Error: Internal Server Error for url: https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%20eq%20datetime'2019-09-10T00:00:00Z'%20and%20EndDate%20eq%20datetime'2019-09-10T01:00:00Z'
1 Solution

nmadhok
Path Finder

We opened a case with Microsoft on 10/31 and the case was resolved by 11/02 after which we were no longer getting 500 Internal Server Error and the Add-On was pulling data once again. Here's the Preliminary Post Incident Review Report from Microsoft related to this incident.

View solution in original post

nmadhok
Path Finder

We opened a case with Microsoft on 10/31 and the case was resolved by 11/02 after which we were no longer getting 500 Internal Server Error and the Add-On was pulling data once again. Here's the Preliminary Post Incident Review Report from Microsoft related to this incident.

poisar
Explorer

Short Update to the Server Error. The error even appears when browsing manual to reports.office365.com.

 

add a \ before the $filter and the error is gone. e.g:

doesnt work: https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate eq datetime'2020-05-28T21:50:04.772888Z' and EndDate eq datetime'2020-05-28T22:50:04.772888Z'

 

works: https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?\$filter=StartDate eq datetime'2020-05-28T21:50:04.772888Z' and EndDate eq datetime'2020-05-28T22:50:04.772888Z'

 

gordo32
Communicator

Adding the backslash into the input_module_ms_o365_message_trace.py at lines 156 & 225 solved this for me (at least today). BTW, using v1.2.1 of the add-on.

Thanks for this solution @poisar !!

jonesy1111
Explorer

Mine Just started working. Not sure what was changed I am reaching out to our MS team to see if they changed anything

0 Karma

tommusgrave
Explorer

The API starting working for us again. MS clearly responsible.

raugugliaro_ao
New Member

Same thing here.

Still have not heard back from MS though.

0 Karma

tommusgrave
Explorer

Seems that this reporting API is totally best effort SLA. Not a nice solution to rely on.

0 Karma

tommusgrave
Explorer

We're getting 500 errors too. When I test with postman to the api without searching I can authenticate ok. Looks like someothing has changed?

https://reports.office365.com/ecp/reportingwebservice/reporting.svc/

I have a support issue open but not much progress there. Would be interesting to know how many people have this issue right now.

0 Karma

nmadhok
Path Finder

@tommusgrave Let us know if you hear back from Microsoft Support. Also upvote the question to keep a count of people affected by this issue

0 Karma

raugugliaro_ao
New Member

I am having the same exact issue. We had the add on working properly for at least 6 months but it started returning an error starting a few days ago.

I have opened a ticket with our Microsoft Support team to see if they can shed some light on this.

Will post my results here when I get more information.

0 Karma

jaivijay_rio
Explorer

Same problem here: i have little to no information on the API changes on o365 reporting service

0 Karma

nmadhok
Path Finder

@raugugliaro_ao Let us know if you hear back from Microsoft Support. Also upvote the question to keep a count of people affected by this issue

0 Karma

jonesy1111
Explorer

I stumbled on this. It looks like Microsoft has made some changes to the API
https://techcommunity.microsoft.com/t5/Office-365-Blog/Announcing-the-General-Availability-of-Micros...

0 Karma

nmadhok
Path Finder

@jonesy1111 The document referenced above says that the MessageTrace method will continue to work as expected and is not impacted by this deprecation

https://docs.microsoft.com/en-us/previous-versions/office/developer/o365-enterprise-developers/jj984...

0 Karma

jonesy1111
Explorer

Oh.. neat, I missed that part. I wonder if the issue is the url format of the query. I am by no means a SME. Just trying to find a solution.

0 Karma

nmadhok
Path Finder

@jonesy1111 Not a problem. Don't think it's an issue with the url format since if you directly hit the API endpoint, it shows the error. Please upvote the question to keep a count of people affected by this issue

0 Karma

jconger
Splunk Employee
Splunk Employee

A 500 error is going to be on the server side - in other words on the API side. The API web service uses basic auth, so it's pretty easy to test with just a browser. Navigate to https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace and log in with an account that has permission. If you get an error there, the add-on will get the same error.

0 Karma

jconger
Splunk Employee
Splunk Employee

Several customers have reported the API is working again on the MSFT side. Your add-on should start catching up since it saved the check point.

If this URL works for you in a browser or Postman, the add-on should be working too -> https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace

0 Karma

nmadhok
Path Finder

@jconger I did that and got the following error:

<m:error xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata">
<m:code/>
<m:message xml:lang="en-US">An error occurred while processing this request.</m:message>
</m:error>
0 Karma

nmadhok
Path Finder

@jconger @lnetto_splunk

0 Karma
Get Updates on the Splunk Community!

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...

DevSecOps: Why You Should Care and How To Get Started

 WATCH NOW In this Tech Talk we will talk about what people mean by DevSecOps and deep dive into the different ...

Introducing Ingest Actions: Filter, Mask, Route, Repeat

WATCH NOW Ingest Actions (IA) is the best new way to easily filter, mask and route your data in Splunk® ...