Microsoft Office 365 Reporting Add-on for Splunk: Python Script Crashes

[bradp1234]

Hello,

I am looking for suggestions on how to troubleshoot the python script crashing. I have experienced my message trace logs stopping two times in the last seven days. To resolve the issue i have to restart Splunk. I would like to continue to use this app, but if it keeps crashing i will have to stop using it. No other apps are installed on the heavy forwarder. I have the debug logs turned on, but they are not showing why the script crashes. The last event before the script crashes is:

2018-11-28 19:19:29,696 DEBUG pid=9007 tid=MainThread file=connectionpool.py:_make_request:400 | https://reports.office365.com:443 "GET /ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%20eq%20datetime'2018-11-27T04%3A59%3A21.366728Z'%20and%20EndDate%20eq%20datetime'2018-11-27T05%3A29%3A21.366728Z'&$skiptoken=27999 HTTP/1.1" 200 None

2018-11-27 14:04:00,778 DEBUG pid=25448 tid=MainThread file=connectionpool.py:_make_request:400 | https://reports.office365.com:443 "GET /ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%20eq%20datetime'2018-11-27T01%3A29%3A21.591658Z'%20and%20EndDate%20eq%20datetime'2018-11-27T01%3A59%3A21.591658Z'&$skiptoken=75999 HTTP/1.1" 200 None

2018-11-25 10:05:20,410 DEBUG pid=21262 tid=MainThread file=connectionpool.py:_make_request:400 | https://reports.office365.com:443 "GET /ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%20eq%20datetime'2018-11-24T21%3A29%3A25.054247Z'%20and%20EndDate%20eq%20datetime'2018-11-24T21%3A59%3A25.054247Z'&$skiptoken=107999 HTTP/1.1" 200 None

Splunk Enterprise version 6.6.7
App version 1.1.0

4 CPU
8 GB RAM
100 GB hard drive

Thanks!

[bradp1234 · Accepted solution]

Final solution was to copy the code out of this app and modify it to send to a HEC endpoint.

[justinbarta_jab]

I've been having the same issue.
Giving this a try
Thanks !!

[andrewzuehlke]

Glad you were able to come up with a solution! Just to clarify, you copied the entire code out of the app, configure it to run on its own (outside of the Splunk environment), and configure it to send information to an HTTP Event Collector?

Thanks!

[bradp1234]

Yes, that is correct.

[andrewzuehlke]

Have you had any luck with this issue? I am in a similar situation, where our message trace logs stop coming in, with the only fix being a restart. The app is running on a Linux Heavy Forwarder with the following specs:

Splunk Enterprise version 7.2.1
App version 1.1.0

16 CPU
32 GB RAM
50 GB hard drive

Thanks!

[bradp1234]

Sorry, we have not had any luck, but we have ruled out some issues. I will update this ticket when we do figure it out.

So far we have added additional debug logs to /opt/splunk/etc/apps/TA-MS_O365_Reporting/bin/input_module_ms_o365_message_trace.py on around line 100 in the get_messages function. The additional debugging shows the response from the API request, but no issues around when the script stops. I am very surprised that the authors have not supported this app better.