We were successfully ingesting O365 management logs using the Splunk O365 add-on. We noticed a drop in events and a really small number of events being forwarded for a day after the drop. After that events stopped coming in completely.
This was a working inputs with logs coming in as expected. No change to a configuration was made to Splunk or O365 whatsoever.
Currently we can see the following error in Splunk each time the input is run. The error appears after the input successfully acquires the access token and calls the management API.
2020-11-30 12:59:34,992 level=ERROR pid=27098 tid=MainThread logger=splunk_ta_o365.modinputs.management_activity pos=utils.py:wrapper:67 | datainput=b'O365_audit_general' start_time=1606741174 | message="Data input was interrupted by an unhandled exception."
Traceback (most recent call last):
File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunksdc/utils.py", line 65, in wrapper
return func(*args, **kwargs)
File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/management_activity.py", line 102, in run
executor.run(adapter)
File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunksdc/batch.py", line 47, in run
for jobs in delegate.discover():
File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/management_activity.py", line 126, in discover
if not subscription.is_enabled(session):
File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/portal.py", line 151, in is_enabled
response = self._perform(session, 'GET', '/subscriptions/list')
File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/portal.py", line 169, in _perform
return self._request(session, method, url, kwargs)
File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/portal.py", line 181, in _request
raise O365PortalError(response)
File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/portal.py", line 26, in __init__
message = str(response.status_code) + ':' + payload
TypeError: can only concatenate str (not "bytes") to str
The same error appears for all add-on inputs (status, message, audit, sharepoint etc.)
Any idea what the problem might be?
