All Apps and Add-ons

Why did O365 Message Trace Logs Stopped Working?

jcleary47
Path Finder

Back on 6/19, one of our inputs for Microsoft Office 365 Reporting Add-on for Splunk stopped working. We actually have two - one of them is still working fine, but one of them no matter what we try/do it just won't resume indexing.

This is the only seemingly relevant error I can see in the _internal logs:

2019-07-02 14:09:02,846 DEBUG pid=107942 tid=MainThread file=connectionpool.py:_make_request:400 | https://reports.office365.com:443 "GET /ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%20eq%20datetime'2019-06-19T15:34:27.739334Z'%20and%20EndDate%20eq%20datetime'2019-06-19T20:34:27.739334Z' HTTP/1.1" 200 216

I found another thread that suggested this Error 400 can be if you have a start date set too far back - well I tried removing the start date all together and it still won't resume indexing anything.

Any suggestions here? I'm hesitant to blow up the app and start over because one of the Message Trace inputs is still working fine. We didn't make any changes to this app or anything at the time it stopped.

Labels (3)

ryan_629
Explorer

Were you ever able to find a solution to this?

0 Karma

mydog8it
Builder

Have you tried restarting Splunk on the DCN and deleting the checkpoint file? I have had the issue and this is the only way I can recover. I am looking at migrating this to one of the methods listed here:
https://www.splunk.com/blog/2017/10/05/splunking-microsoft-cloud-data-part-3.html

0 Karma

jcleary47
Path Finder

When you say disabled on the DCN, what does that mean? We got the input to start working again by creating it new from scratch and then deleting the old one. But it hung up again shortly after and is stuck on 7/1.

Where is the checkpoint file that you are deleting? Would be curious to see if we can try that instead of deleting and recreating from scratch again.

0 Karma

nivin0910
Engager

@mydog8it

Would also like to know where this checkpoint file is located? Have been having ongoing issues, and would like to delete it. A restart or recreation of the inputs file doesn't work for me.

Unless its under $SPLUNK_HOME/var/lib/splunk/modinputs
Then I have nothing inside my dir.

0 Karma

Esky73
Builder

We also have similar issues where the ingestion randomly stops - there is nothing in the logs we've found of use to triage.

Anything from the app devs ? thx

0 Karma

jcleary47
Path Finder

No, have not seen any updates to the app or comments from the developers yet. There a way to contact them?

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...