All Apps and Add-ons

Microsoft Office 365 Reporting Add-on for Splunk not pulling data - exiting with 500 Internal Server Error

nmadhok
Path Finder

Installed and configured Microsoft Office 365 Reporting Add-on for Splunk but it doesn't seem to be pulling any data. Here's the error we see in the ta_ms_o365_reporting_ms_o365_message_trace.log file:

2019-10-30 16:21:16,836 INFO pid=40891 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-10-30 16:21:17,674 INFO pid=40891 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-10-30 16:21:19,141 INFO pid=40891 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-10-30 16:21:20,569 INFO pid=40891 tid=MainThread file=splunk_rest_client.py:_request_handler:100 | Use HTTP connection pooling
2019-10-30 16:21:20,570 DEBUG pid=40891 tid=MainThread file=binding.py:get:664 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/config/TA_MS_O365_Reporting_checkpointer (body: {})
2019-10-30 16:21:20,571 INFO pid=40891 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-10-30 16:21:20,576 DEBUG pid=40891 tid=MainThread file=connectionpool.py:_make_request:387 | "GET /servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/config/TA_MS_O365_Reporting_checkpointer HTTP/1.1" 200 5516
2019-10-30 16:21:20,577 DEBUG pid=40891 tid=MainThread file=binding.py:new_f:71 | Operation took 0:00:00.006580
2019-10-30 16:21:20,577 DEBUG pid=40891 tid=MainThread file=binding.py:get:664 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/config/ (body: {'count': -1, 'search': 'TA_MS_O365_Reporting_checkpointer', 'offset': 0})
2019-10-30 16:21:20,580 DEBUG pid=40891 tid=MainThread file=connectionpool.py:_make_request:387 | "GET /servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/config/?count=-1&search=TA_MS_O365_Reporting_checkpointer&offset=0 HTTP/1.1" 200 7417
2019-10-30 16:21:20,580 DEBUG pid=40891 tid=MainThread file=binding.py:new_f:71 | Operation took 0:00:00.003192
2019-10-30 16:21:20,583 DEBUG pid=40891 tid=MainThread file=binding.py:get:664 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/data/TA_MS_O365_Reporting_checkpointer/o365_message_trace_obj_checkpoint (body: {})
2019-10-30 16:21:20,585 DEBUG pid=40891 tid=MainThread file=connectionpool.py:_make_request:387 | "GET /servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/data/TA_MS_O365_Reporting_checkpointer/o365_message_trace_obj_checkpoint HTTP/1.1" 404 140
2019-10-30 16:21:20,587 DEBUG pid=40891 tid=MainThread file=base_modinput.py:log_debug:286 | Start date: 2019-09-10 00:00:00, End date: 2019-09-10 01:00:00
2019-10-30 16:21:20,587 DEBUG pid=40891 tid=MainThread file=base_modinput.py:log_debug:286 | Endpoint URL: https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate eq datetime'2019-09-10T00:00:00Z' and EndDate eq datetime'2019-09-10T01:00:00Z'
2019-10-30 16:21:20,587 INFO pid=40891 tid=MainThread file=setup_util.py:log_info:114 | Proxy is not enabled!
2019-10-30 16:21:20,596 DEBUG pid=40891 tid=MainThread file=connectionpool.py:_new_conn:809 | Starting new HTTPS connection (1): reports.office365.com
2019-10-30 16:21:20,976 DEBUG pid=40891 tid=MainThread file=connectionpool.py:_make_request:400 | https://reports.office365.com:443 "GET /ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%20eq%20datetime'2019-09-10T00:00:00Z'%20and%20EndDate%20eq%20datetime'2019-09-10T01:00:00Z' HTTP/1.1" 500 113
2019-10-30 16:21:20,979 ERROR pid=40891 tid=MainThread file=base_modinput.py:log_error:307 | HTTP Request error: 500 Server Error: Internal Server Error for url: https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%20eq%20datetime'2019-09-10T00:00:00Z'%20and%20EndDate%20eq%20datetime'2019-09-10T01:00:00Z'
1 Solution

nmadhok
Path Finder

We opened a case with Microsoft on 10/31 and the case was resolved by 11/02 after which we were no longer getting 500 Internal Server Error and the Add-On was pulling data once again. Here's the Preliminary Post Incident Review Report from Microsoft related to this incident.

View solution in original post

mik3y
Path Finder

I can confirm this was affecting version 2.0.1 and 2.0.0 and appeared to break around 7/Sept/2022 for us.

Modifying the following two files resolved the issue

(grep "microsoft_trace_url =" /opt/splunk/etc/apps/TA-MS_O365_Reporting/bin/*.py)

OR

/opt/splunk/etc/apps/TA-MS_O365_Reporting/bin/ms_o365_message_trace_oauth.py

/opt/splunk/etc/apps/TA-MS_O365_Reporting/bin/ms_o365_message_trace.py

Changing the URL (to include a backslash) from

https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?$filter

to

https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?\$filter

Will lodge a splunk support ticket now.

Cheers

0 Karma

mik3y
Path Finder

I guess Splunk don't support Splunk or Splunk Works developed apps now.

haraksin
Path Finder

I opened a case with microsoft for this error - the issue for me was that the docs (both on splunkbase and in the Microsoft docs article referenced by Splunkbase) say that the Message Trace API can gather data up to 30 days prior. This is incorrect. The correct documentation can be found here, which shows that only a max of 10 days prior is allowed: https://docs.microsoft.com/en-us/powershell/module/exchange/get-messagetrace?redirectedfrom=MSDN&vie...

0 Karma

nmadhok
Path Finder

We opened a case with Microsoft on 10/31 and the case was resolved by 11/02 after which we were no longer getting 500 Internal Server Error and the Add-On was pulling data once again. Here's the Preliminary Post Incident Review Report from Microsoft related to this incident.

poisar
Explorer

Short Update to the Server Error. The error even appears when browsing manual to reports.office365.com.

 

add a \ before the $filter and the error is gone. e.g:

doesnt work: https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate eq datetime'2020-05-28T21:50:04.772888Z' and EndDate eq datetime'2020-05-28T22:50:04.772888Z'

 

works: https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?\$filter=StartDate eq datetime'2020-05-28T21:50:04.772888Z' and EndDate eq datetime'2020-05-28T22:50:04.772888Z'

 

gordo32
Communicator

Adding the backslash into the input_module_ms_o365_message_trace.py at lines 156 & 225 solved this for me (at least today). BTW, using v1.2.1 of the add-on.

Thanks for this solution @poisar !!

jonesy1111
Explorer

Mine Just started working. Not sure what was changed I am reaching out to our MS team to see if they changed anything

0 Karma

tommusgrave
Explorer

The API starting working for us again. MS clearly responsible.

raugugliaro_ao
New Member

Same thing here.

Still have not heard back from MS though.

0 Karma

tommusgrave
Explorer

Seems that this reporting API is totally best effort SLA. Not a nice solution to rely on.

0 Karma

tommusgrave
Explorer

We're getting 500 errors too. When I test with postman to the api without searching I can authenticate ok. Looks like someothing has changed?

https://reports.office365.com/ecp/reportingwebservice/reporting.svc/

I have a support issue open but not much progress there. Would be interesting to know how many people have this issue right now.

0 Karma

nmadhok
Path Finder

@tommusgrave Let us know if you hear back from Microsoft Support. Also upvote the question to keep a count of people affected by this issue

0 Karma

raugugliaro_ao
New Member

I am having the same exact issue. We had the add on working properly for at least 6 months but it started returning an error starting a few days ago.

I have opened a ticket with our Microsoft Support team to see if they can shed some light on this.

Will post my results here when I get more information.

0 Karma

jaivijay_rio
Explorer

Same problem here: i have little to no information on the API changes on o365 reporting service

0 Karma

nmadhok
Path Finder

@raugugliaro_ao Let us know if you hear back from Microsoft Support. Also upvote the question to keep a count of people affected by this issue

0 Karma

jonesy1111
Explorer

I stumbled on this. It looks like Microsoft has made some changes to the API
https://techcommunity.microsoft.com/t5/Office-365-Blog/Announcing-the-General-Availability-of-Micros...

0 Karma

nmadhok
Path Finder

@jonesy1111 The document referenced above says that the MessageTrace method will continue to work as expected and is not impacted by this deprecation

https://docs.microsoft.com/en-us/previous-versions/office/developer/o365-enterprise-developers/jj984...

0 Karma

jonesy1111
Explorer

Oh.. neat, I missed that part. I wonder if the issue is the url format of the query. I am by no means a SME. Just trying to find a solution.

0 Karma

nmadhok
Path Finder

@jonesy1111 Not a problem. Don't think it's an issue with the url format since if you directly hit the API endpoint, it shows the error. Please upvote the question to keep a count of people affected by this issue

0 Karma

jconger
Splunk Employee
Splunk Employee

A 500 error is going to be on the server side - in other words on the API side. The API web service uses basic auth, so it's pretty easy to test with just a browser. Navigate to https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace and log in with an account that has permission. If you get an error there, the add-on will get the same error.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...