All Apps and Add-ons

Microsoft Office 365 Reporting Add-on for Splunk: Python Script Crashes

bradp1234
Path Finder

Hello,

I am looking for suggestions on how to troubleshoot the python script crashing. I have experienced my message trace logs stopping two times in the last seven days. To resolve the issue i have to restart Splunk. I would like to continue to use this app, but if it keeps crashing i will have to stop using it. No other apps are installed on the heavy forwarder. I have the debug logs turned on, but they are not showing why the script crashes. The last event before the script crashes is:

2018-11-28 19:19:29,696 DEBUG pid=9007 tid=MainThread file=connectionpool.py:_make_request:400 | https://reports.office365.com:443 "GET /ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%20eq%20datetime'2018-11-27T04%3A59%3A21.366728Z'%20and%20EndDate%20eq%20datetime'2018-11-27T05%3A29%3A21.366728Z'&$skiptoken=27999 HTTP/1.1" 200 None

2018-11-27 14:04:00,778 DEBUG pid=25448 tid=MainThread file=connectionpool.py:_make_request:400 | https://reports.office365.com:443 "GET /ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%20eq%20datetime'2018-11-27T01%3A29%3A21.591658Z'%20and%20EndDate%20eq%20datetime'2018-11-27T01%3A59%3A21.591658Z'&$skiptoken=75999 HTTP/1.1" 200 None

2018-11-25 10:05:20,410 DEBUG pid=21262 tid=MainThread file=connectionpool.py:_make_request:400 | https://reports.office365.com:443 "GET /ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%20eq%20datetime'2018-11-24T21%3A29%3A25.054247Z'%20and%20EndDate%20eq%20datetime'2018-11-24T21%3A59%3A25.054247Z'&$skiptoken=107999 HTTP/1.1" 200 None

alt text

Splunk Enterprise version 6.6.7
App version 1.1.0

4 CPU
8 GB RAM
100 GB hard drive

Thanks!

1 Solution

bradp1234
Path Finder

Final solution was to copy the code out of this app and modify it to send to a HEC endpoint.

View solution in original post

0 Karma

bradp1234
Path Finder

Final solution was to copy the code out of this app and modify it to send to a HEC endpoint.

View solution in original post

0 Karma

justinbarta_jab
Engager

I've been having the same issue.
Giving this a try
Thanks !!

0 Karma

andrewzuehlke
Explorer

Glad you were able to come up with a solution! Just to clarify, you copied the entire code out of the app, configure it to run on its own (outside of the Splunk environment), and configure it to send information to an HTTP Event Collector?

Thanks!

0 Karma

bradp1234
Path Finder

Yes, that is correct.

0 Karma

andrewzuehlke
Explorer

Have you had any luck with this issue? I am in a similar situation, where our message trace logs stop coming in, with the only fix being a restart. The app is running on a Linux Heavy Forwarder with the following specs:

Splunk Enterprise version 7.2.1
App version 1.1.0

16 CPU
32 GB RAM
50 GB hard drive

Thanks!

0 Karma

bradp1234
Path Finder

Sorry, we have not had any luck, but we have ruled out some issues. I will update this ticket when we do figure it out.

So far we have added additional debug logs to /opt/splunk/etc/apps/TA-MS_O365_Reporting/bin/input_module_ms_o365_message_trace.py on around line 100 in the get_messages function. The additional debugging shows the response from the API request, but no issues around when the script stops. I am very surprised that the authors have not supported this app better.

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!