Splunk Cloud Platform

O365 Add-on sudden stop forwarding of logs, "Data input was interrupted by an unhandled exception." error.

konstr
Path Finder

We were successfully ingesting O365 management logs using the Splunk O365 add-on. We noticed a drop in events and a really small number of events being forwarded for a day after the drop. After that events stopped coming in completely.

This was a working inputs with logs coming in as expected. No change to a configuration was made to Splunk or O365 whatsoever.

 

Currently we can see the following error in Splunk each time the input is run. The error appears after the input successfully acquires the access token and calls the management API.

 

 

 

2020-11-30 12:59:34,992 level=ERROR pid=27098 tid=MainThread logger=splunk_ta_o365.modinputs.management_activity pos=utils.py:wrapper:67 | datainput=b'O365_audit_general' start_time=1606741174 | message="Data input was interrupted by an unhandled exception." 
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunksdc/utils.py", line 65, in wrapper
    return func(*args, **kwargs)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/management_activity.py", line 102, in run
    executor.run(adapter)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunksdc/batch.py", line 47, in run
    for jobs in delegate.discover():
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/management_activity.py", line 126, in discover
    if not subscription.is_enabled(session):
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/portal.py", line 151, in is_enabled
    response = self._perform(session, 'GET', '/subscriptions/list')
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/portal.py", line 169, in _perform
    return self._request(session, method, url, kwargs)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/portal.py", line 181, in _request
    raise O365PortalError(response)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/portal.py", line 26, in __init__
    message = str(response.status_code) + ':' + payload
TypeError: can only concatenate str (not "bytes") to str

 

 

 

The same error appears for all add-on inputs (status, message, audit, sharepoint etc.)

 

Any idea what the problem might be?

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...