Splunk Cloud Platform

O365 Add-on sudden stop forwarding of logs, "Data input was interrupted by an unhandled exception." error.

konstr
Path Finder

We were successfully ingesting O365 management logs using the Splunk O365 add-on. We noticed a drop in events and a really small number of events being forwarded for a day after the drop. After that events stopped coming in completely.

This was a working inputs with logs coming in as expected. No change to a configuration was made to Splunk or O365 whatsoever.

 

Currently we can see the following error in Splunk each time the input is run. The error appears after the input successfully acquires the access token and calls the management API.

 

 

 

2020-11-30 12:59:34,992 level=ERROR pid=27098 tid=MainThread logger=splunk_ta_o365.modinputs.management_activity pos=utils.py:wrapper:67 | datainput=b'O365_audit_general' start_time=1606741174 | message="Data input was interrupted by an unhandled exception." 
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunksdc/utils.py", line 65, in wrapper
    return func(*args, **kwargs)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/management_activity.py", line 102, in run
    executor.run(adapter)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunksdc/batch.py", line 47, in run
    for jobs in delegate.discover():
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/management_activity.py", line 126, in discover
    if not subscription.is_enabled(session):
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/portal.py", line 151, in is_enabled
    response = self._perform(session, 'GET', '/subscriptions/list')
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/portal.py", line 169, in _perform
    return self._request(session, method, url, kwargs)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/portal.py", line 181, in _request
    raise O365PortalError(response)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/portal.py", line 26, in __init__
    message = str(response.status_code) + ':' + payload
TypeError: can only concatenate str (not "bytes") to str

 

 

 

The same error appears for all add-on inputs (status, message, audit, sharepoint etc.)

 

Any idea what the problem might be?

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...