Splunk Cloud Platform

O365 Add-on sudden stop forwarding of logs, "Data input was interrupted by an unhandled exception." error.

konstr
Path Finder

We were successfully ingesting O365 management logs using the Splunk O365 add-on. We noticed a drop in events and a really small number of events being forwarded for a day after the drop. After that events stopped coming in completely.

This was a working inputs with logs coming in as expected. No change to a configuration was made to Splunk or O365 whatsoever.

 

Currently we can see the following error in Splunk each time the input is run. The error appears after the input successfully acquires the access token and calls the management API.

 

 

 

2020-11-30 12:59:34,992 level=ERROR pid=27098 tid=MainThread logger=splunk_ta_o365.modinputs.management_activity pos=utils.py:wrapper:67 | datainput=b'O365_audit_general' start_time=1606741174 | message="Data input was interrupted by an unhandled exception." 
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunksdc/utils.py", line 65, in wrapper
    return func(*args, **kwargs)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/management_activity.py", line 102, in run
    executor.run(adapter)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunksdc/batch.py", line 47, in run
    for jobs in delegate.discover():
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/management_activity.py", line 126, in discover
    if not subscription.is_enabled(session):
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/portal.py", line 151, in is_enabled
    response = self._perform(session, 'GET', '/subscriptions/list')
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/portal.py", line 169, in _perform
    return self._request(session, method, url, kwargs)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/portal.py", line 181, in _request
    raise O365PortalError(response)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/portal.py", line 26, in __init__
    message = str(response.status_code) + ':' + payload
TypeError: can only concatenate str (not "bytes") to str

 

 

 

The same error appears for all add-on inputs (status, message, audit, sharepoint etc.)

 

Any idea what the problem might be?

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!