Microsoft Office 365 Reporting Add-on for Splunk not pulling data - exiting with 500 Internal Server Error

[nmadhok]

Installed and configured Microsoft Office 365 Reporting Add-on for Splunk but it doesn't seem to be pulling any data. Here's the error we see in the ta_ms_o365_reporting_ms_o365_message_trace.log file:

2019-10-30 16:21:16,836 INFO pid=40891 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-10-30 16:21:17,674 INFO pid=40891 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-10-30 16:21:19,141 INFO pid=40891 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-10-30 16:21:20,569 INFO pid=40891 tid=MainThread file=splunk_rest_client.py:_request_handler:100 | Use HTTP connection pooling
2019-10-30 16:21:20,570 DEBUG pid=40891 tid=MainThread file=binding.py:get:664 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/config/TA_MS_O365_Reporting_checkpointer (body: {})
2019-10-30 16:21:20,571 INFO pid=40891 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-10-30 16:21:20,576 DEBUG pid=40891 tid=MainThread file=connectionpool.py:_make_request:387 | "GET /servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/config/TA_MS_O365_Reporting_checkpointer HTTP/1.1" 200 5516
2019-10-30 16:21:20,577 DEBUG pid=40891 tid=MainThread file=binding.py:new_f:71 | Operation took 0:00:00.006580
2019-10-30 16:21:20,577 DEBUG pid=40891 tid=MainThread file=binding.py:get:664 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/config/ (body: {'count': -1, 'search': 'TA_MS_O365_Reporting_checkpointer', 'offset': 0})
2019-10-30 16:21:20,580 DEBUG pid=40891 tid=MainThread file=connectionpool.py:_make_request:387 | "GET /servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/config/?count=-1&search=TA_MS_O365_Reporting_checkpointer&offset=0 HTTP/1.1" 200 7417
2019-10-30 16:21:20,580 DEBUG pid=40891 tid=MainThread file=binding.py:new_f:71 | Operation took 0:00:00.003192
2019-10-30 16:21:20,583 DEBUG pid=40891 tid=MainThread file=binding.py:get:664 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/data/TA_MS_O365_Reporting_checkpointer/o365_message_trace_obj_checkpoint (body: {})
2019-10-30 16:21:20,585 DEBUG pid=40891 tid=MainThread file=connectionpool.py:_make_request:387 | "GET /servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/data/TA_MS_O365_Reporting_checkpointer/o365_message_trace_obj_checkpoint HTTP/1.1" 404 140
2019-10-30 16:21:20,587 DEBUG pid=40891 tid=MainThread file=base_modinput.py:log_debug:286 | Start date: 2019-09-10 00:00:00, End date: 2019-09-10 01:00:00
2019-10-30 16:21:20,587 DEBUG pid=40891 tid=MainThread file=base_modinput.py:log_debug:286 | Endpoint URL: https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate eq datetime'2019-09-10T00:00:00Z' and EndDate eq datetime'2019-09-10T01:00:00Z'
2019-10-30 16:21:20,587 INFO pid=40891 tid=MainThread file=setup_util.py:log_info:114 | Proxy is not enabled!
2019-10-30 16:21:20,596 DEBUG pid=40891 tid=MainThread file=connectionpool.py:_new_conn:809 | Starting new HTTPS connection (1): reports.office365.com
2019-10-30 16:21:20,976 DEBUG pid=40891 tid=MainThread file=connectionpool.py:_make_request:400 | https://reports.office365.com:443 "GET /ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%20eq%20datetime'2019-09-10T00:00:00Z'%20and%20EndDate%20eq%20datetime'2019-09-10T01:00:00Z' HTTP/1.1" 500 113
2019-10-30 16:21:20,979 ERROR pid=40891 tid=MainThread file=base_modinput.py:log_error:307 | HTTP Request error: 500 Server Error: Internal Server Error for url: https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%20eq%20datetime'2019-09-10T00:00:00Z'%20and%20EndDate%20eq%20datetime'2019-09-10T01:00:00Z'

[jconger]

Several customers have reported the API is working again on the MSFT side. Your add-on should start catching up since it saved the check point.

If this URL works for you in a browser or Postman, the add-on should be working too -> https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace

[nmadhok]

@jconger I did that and got the following error:

<m:error xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata">
<m:code/>
<m:message xml:lang="en-US">An error occurred while processing this request.</m:message>
</m:error>

[nmadhok]

@jconger @lnetto_splunk

[jonesy1111]

Im getting the same error. I tried creating a new input, as well as a complete reinstall of the add-on.
I logged into the admin panel with the user account to verify permissions. Would love a solution to this.