topic Re: Microsoft Office 365 Reporting Add-on for Splunk not pulling data - exiting with 500 Internal Server Error in All Apps and Add-ons

Microsoft Office 365 Reporting Add-on for Splunk not pulling data - exiting with 500 Internal Server Error

nmadhok — Wed, 30 Sep 2020 02:47:06 GMT

Installed and configured Microsoft Office 365 Reporting Add-on for Splunk but it doesn't seem to be pulling any data. Here's the error we see in the ta_ms_o365_reporting_ms_o365_message_trace.log file:

2019-10-30 16:21:16,836 INFO pid=40891 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-10-30 16:21:17,674 INFO pid=40891 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-10-30 16:21:19,141 INFO pid=40891 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-10-30 16:21:20,569 INFO pid=40891 tid=MainThread file=splunk_rest_client.py:_request_handler:100 | Use HTTP connection pooling
2019-10-30 16:21:20,570 DEBUG pid=40891 tid=MainThread file=binding.py:get:664 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/config/TA_MS_O365_Reporting_checkpointer (body: {})
2019-10-30 16:21:20,571 INFO pid=40891 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-10-30 16:21:20,576 DEBUG pid=40891 tid=MainThread file=connectionpool.py:_make_request:387 | "GET /servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/config/TA_MS_O365_Reporting_checkpointer HTTP/1.1" 200 5516
2019-10-30 16:21:20,577 DEBUG pid=40891 tid=MainThread file=binding.py:new_f:71 | Operation took 0:00:00.006580
2019-10-30 16:21:20,577 DEBUG pid=40891 tid=MainThread file=binding.py:get:664 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/config/ (body: {'count': -1, 'search': 'TA_MS_O365_Reporting_checkpointer', 'offset': 0})
2019-10-30 16:21:20,580 DEBUG pid=40891 tid=MainThread file=connectionpool.py:_make_request:387 | "GET /servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/config/?count=-1&search=TA_MS_O365_Reporting_checkpointer&offset=0 HTTP/1.1" 200 7417
2019-10-30 16:21:20,580 DEBUG pid=40891 tid=MainThread file=binding.py:new_f:71 | Operation took 0:00:00.003192
2019-10-30 16:21:20,583 DEBUG pid=40891 tid=MainThread file=binding.py:get:664 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/data/TA_MS_O365_Reporting_checkpointer/o365_message_trace_obj_checkpoint (body: {})
2019-10-30 16:21:20,585 DEBUG pid=40891 tid=MainThread file=connectionpool.py:_make_request:387 | "GET /servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/data/TA_MS_O365_Reporting_checkpointer/o365_message_trace_obj_checkpoint HTTP/1.1" 404 140
2019-10-30 16:21:20,587 DEBUG pid=40891 tid=MainThread file=base_modinput.py:log_debug:286 | Start date: 2019-09-10 00:00:00, End date: 2019-09-10 01:00:00
2019-10-30 16:21:20,587 DEBUG pid=40891 tid=MainThread file=base_modinput.py:log_debug:286 | Endpoint URL: https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate eq datetime'2019-09-10T00:00:00Z' and EndDate eq datetime'2019-09-10T01:00:00Z'
2019-10-30 16:21:20,587 INFO pid=40891 tid=MainThread file=setup_util.py:log_info:114 | Proxy is not enabled!
2019-10-30 16:21:20,596 DEBUG pid=40891 tid=MainThread file=connectionpool.py:_new_conn:809 | Starting new HTTPS connection (1): reports.office365.com
2019-10-30 16:21:20,976 DEBUG pid=40891 tid=MainThread file=connectionpool.py:_make_request:400 | https://reports.office365.com:443 "GET /ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%20eq%20datetime'2019-09-10T00:00:00Z'%20and%20EndDate%20eq%20datetime'2019-09-10T01:00:00Z' HTTP/1.1" 500 113
2019-10-30 16:21:20,979 ERROR pid=40891 tid=MainThread file=base_modinput.py:log_error:307 | HTTP Request error: 500 Server Error: Internal Server Error for url: https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%20eq%20datetime'2019-09-10T00:00:00Z'%20and%20EndDate%20eq%20datetime'2019-09-10T01:00:00Z'

Re: Microsoft Office 365 Reporting Add-on for Splunk not pulling data - exiting with 500 Internal Server Error

nmadhok — Thu, 31 Oct 2019 13:55:38 GMT

@jconger @lnetto_splunk

Re: Microsoft Office 365 Reporting Add-on for Splunk not pulling data - exiting with 500 Internal Server Error

jonesy1111 — Thu, 31 Oct 2019 15:32:22 GMT

Im getting the same error. I tried creating a new input, as well as a complete reinstall of the add-on.
I logged into the admin panel with the user account to verify permissions. Would love a solution to this.

Re: Microsoft Office 365 Reporting Add-on for Splunk not pulling data - exiting with 500 Internal Server Error

jconger — Thu, 31 Oct 2019 15:47:00 GMT

A 500 error is going to be on the server side - in other words on the API side. The API web service uses basic auth, so it's pretty easy to test with just a browser. Navigate to https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace and log in with an account that has permission. If you get an error there, the add-on will get the same error.

Re: Microsoft Office 365 Reporting Add-on for Splunk not pulling data - exiting with 500 Internal Server Error

nmadhok — Thu, 31 Oct 2019 16:03:11 GMT

@jconger I did that and got the following error:

<m:error xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata">
<m:code/>
<m:message xml:lang="en-US">An error occurred while processing this request.</m:message>
</m:error>

Re: Microsoft Office 365 Reporting Add-on for Splunk not pulling data - exiting with 500 Internal Server Error

jonesy1111 — Thu, 31 Oct 2019 16:11:34 GMT

I stumbled on this. It looks like Microsoft has made some changes to the API
https://techcommunity.microsoft.com/t5/Office-365-Blog/Announcing-the-General-Availability-of-Microsoft-Graph-reporting/ba-p/137838

Re: Microsoft Office 365 Reporting Add-on for Splunk not pulling data - exiting with 500 Internal Server Error

nmadhok — Thu, 31 Oct 2019 16:25:31 GMT

@jonesy1111 The document referenced above says that the MessageTrace method will continue to work as expected and is not impacted by this deprecation

https://docs.microsoft.com/en-us/previous-versions/office/developer/o365-enterprise-developers/jj984335(v=office.15)

Re: Microsoft Office 365 Reporting Add-on for Splunk not pulling data - exiting with 500 Internal Server Error

jonesy1111 — Thu, 31 Oct 2019 16:28:20 GMT

Oh.. neat, I missed that part. I wonder if the issue is the url format of the query. I am by no means a SME. Just trying to find a solution.

Re: Microsoft Office 365 Reporting Add-on for Splunk not pulling data - exiting with 500 Internal Server Error

raugugliaro_ao — Thu, 31 Oct 2019 18:01:25 GMT

I am having the same exact issue. We had the add on working properly for at least 6 months but it started returning an error starting a few days ago.

I have opened a ticket with our Microsoft Support team to see if they can shed some light on this.

Will post my results here when I get more information.

Re: Microsoft Office 365 Reporting Add-on for Splunk not pulling data - exiting with 500 Internal Server Error

tommusgrave — Thu, 31 Oct 2019 18:25:42 GMT

We're getting 500 errors too. When I test with postman to the api without searching I can authenticate ok. Looks like someothing has changed?

https://reports.office365.com/ecp/reportingwebservice/reporting.svc/

I have a support issue open but not much progress there. Would be interesting to know how many people have this issue right now.

Re: Microsoft Office 365 Reporting Add-on for Splunk not pulling data - exiting with 500 Internal Server Error

nmadhok — Thu, 31 Oct 2019 18:39:25 GMT

@tommusgrave Let us know if you hear back from Microsoft Support. Also upvote the question to keep a count of people affected by this issue

Re: Microsoft Office 365 Reporting Add-on for Splunk not pulling data - exiting with 500 Internal Server Error

nmadhok — Thu, 31 Oct 2019 18:51:32 GMT

@raugugliaro_ao Let us know if you hear back from Microsoft Support. Also upvote the question to keep a count of people affected by this issue

Re: Microsoft Office 365 Reporting Add-on for Splunk not pulling data - exiting with 500 Internal Server Error

nmadhok — Thu, 31 Oct 2019 18:53:21 GMT

@jonesy1111 Not a problem. Don't think it's an issue with the url format since if you directly hit the API endpoint, it shows the error. Please upvote the question to keep a count of people affected by this issue

Re: Microsoft Office 365 Reporting Add-on for Splunk not pulling data - exiting with 500 Internal Server Error

jaivijay_rio — Thu, 31 Oct 2019 20:25:49 GMT

Same problem here: i have little to no information on the API changes on o365 reporting service

Re: Microsoft Office 365 Reporting Add-on for Splunk not pulling data - exiting with 500 Internal Server Error

tommusgrave — Fri, 01 Nov 2019 15:12:01 GMT

The API starting working for us again. MS clearly responsible.

Re: Microsoft Office 365 Reporting Add-on for Splunk not pulling data - exiting with 500 Internal Server Error

jconger — Fri, 01 Nov 2019 15:26:32 GMT

Several customers have reported the API is working again on the MSFT side. Your add-on should start catching up since it saved the check point.

If this URL works for you in a browser or Postman, the add-on should be working too -> https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace

Re: Microsoft Office 365 Reporting Add-on for Splunk not pulling data - exiting with 500 Internal Server Error

raugugliaro_ao — Fri, 01 Nov 2019 15:51:41 GMT

Same thing here.

Still have not heard back from MS though.

Re: Microsoft Office 365 Reporting Add-on for Splunk not pulling data - exiting with 500 Internal Server Error

tommusgrave — Fri, 01 Nov 2019 18:03:23 GMT

Seems that this reporting API is totally best effort SLA. Not a nice solution to rely on.

Re: Microsoft Office 365 Reporting Add-on for Splunk not pulling data - exiting with 500 Internal Server Error

jonesy1111 — Tue, 05 Nov 2019 17:12:02 GMT

Mine Just started working. Not sure what was changed I am reaching out to our MS team to see if they changed anything

Re: Microsoft Office 365 Reporting Add-on for Splunk not pulling data - exiting with 500 Internal Server Error

nmadhok — Thu, 05 Dec 2019 13:23:24 GMT

We opened a case with Microsoft on 10/31 and the case was resolved by 11/02 after which we were no longer getting 500 Internal Server Error and the Add-On was pulling data once again. Here's the Preliminary Post Incident Review Report from Microsoft related to this incident.