Problem fetching O365 MessageTrace Logs

[pedro_barreiros]

Problem

We are having trouble fetching MessageTrace logs using the "Splunk Add-on for Microsoft Office 365 (Version 4.2.1)" app.

Every time we configure the input with a specific start_date, it ingests a block of logs from a start_date until an end_date. However, after ingesting the first block of logs, the script throws an error. The following times it tries to ingest the logs, it uses the same start and end dates.

Here are the errors we are seeing:

First it collects the logs:

2023-04-18 11:02:21,951 level=INFO pid=22264 tid=MainThread logger=splunk_ta_o365.modinputs.message_trace pos=__init__.py:_get_events_continuous:204 | datainput=b'MessageTraceLogs' start_time=1681812136 | message="Collecting the data between Start date: 2023-04-17 01:00:00, End date: 2023-04-17 02:00:00"

And then it throws an error:

2023-04-18 11:02:24,259 level=ERROR pid=22264 tid=MainThread logger=splunk_ta_o365.modinputs.message_trace pos=__init__.py:_get_messages:253 | datainput=b'MessageTraceLogs' start_time=1681812136 | message="HTTP Request error: 401 Client Error:  for url: https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%20eq%20datetime'2023-04-17T01%3A00%3A00Z'%20and%20EndDate%20eq%20datetime'2023-04-17T02%3A00%3A00Z'&$skiptoken=1999" stack_info=True
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/message_trace/__init__.py", line 246, in _get_messages
    response.raise_for_status()
  File "/opt/splunk/etc/apps/splunk_ta_o365/lib/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error:  for url: https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%20eq%20datetime'2023-04-17T01%3A00%3A00Z'%20and%20EndDate%20eq%20datetime'2023-04-17T02%3A00%3A00Z'&$skiptoken=1999
2023-04-18 11:02:24,260 level=ERROR pid=22264 tid=MainThread logger=splunk_ta_o365.modinputs.message_trace pos=__init__.py:run:359 | datainput=b'MessageTraceLogs' start_time=1681812136 | message="An error occurred while collecting data" stack_info=True
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/message_trace/__init__.py", line 354, in run
    self._collect_events(app)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/message_trace/__init__.py", line 144, in _collect_events
    self._get_events_continuous(app)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/message_trace/__init__.py", line 215, in _get_events_continuous
    self._process_messages(start_date, end_date)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/message_trace/__init__.py", line 306, in _process_messages
    message_response = self._get_messages(nextLink)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/message_trace/__init__.py", line 254, in _get_messages
    return messages
UnboundLocalError: local variable 'messages' referenced before assignment
2023-04-18 11:02:24,265 level=INFO pid=22264 tid=MainThread logger=splunksdc.collector pos=collector.py:run:270 |  | message="Modular input exited."

Our inputs are configured as follows:

[splunk_ta_o365_message_trace://MessageTraceLogs]
delay_throttle = 1440
index = OUR_INDEX
input_mode = continuously_monitor
interval = 300
query_window_size = 60
start_date_time = 2023-04-17T00:00:00
tenant_name = OUR_TENANT
start_by_shell = false
disabled = 0

We have been without continuous monitoring for more than a week.
Thanks for the Help!

[VatsalJagani]

@pedro_barreiros - I would say create a new input with a continuous monitor from UI. Do not update the existing one, create totally new input. And try.

If that does not work than you can reach-out to Splunk Support with the new error that you receive.

I hope this helps!! Upvote if it does!!

[raymondmelendez]

 I am experiencing the same issue and it started around a week ago as well, so far, I have recreated the input with no luck

[VatsalJagani]

@raymondmelendez - Try Splunk Support.