O365 Secret Key expiration doesn't generate expected return code

[gordo32]

I just ran into an issue where the O365 app created for log collection had it's Secret Key expire. According to http://docs.splunk.com/Documentation/AddOns/released/MSO365/Troubleshooting there should be a 401 or 500 error generated. But in fact, what is generated it an unhandled exception (running Add-on for O365 2.0.0, but no reason that it would be fixed in current 2.0.3 version):

2021-01-15 16:26:09,016 level=ERROR pid=12670 tid=MainThread logger=splunk_ta_o365.modinputs.management_activity pos=utils.py:wrapper:67 | start_time=1610745965 datainput="Audit_AzureActiveDirectory" | message="Data input was interrupted by an unhandled exception." Traceback (most recent call last): File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunksdc/utils.py", line 65, in wrapper return func(*args, **kwargs) File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/management_activity.py", line 102, in run executor.run(adapter) File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunksdc/batch.py", line 47, in run for jobs in delegate.discover(): File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/management_activity.py", line 125, in discover self.token.auth(session) File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/token.py", line 56, in auth self._token = self._policy(self._resource, session) File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/token.py", line 37, in __call_ return self.portal.get_token_by_psk(self._client_id, self._client_secret, resource, session) File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/portal.py", line 98, in get_token_by_psk raise O365PortalError(response) File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/portal.py", line 31, in __init_ self._code = data['error']['code'] TypeError: string indices must be integers

I already added the comment to the documentation and they suggested I post it here as well. For now, I have an ugly alert/query in case this happens again:

index=_internal sourcetype="splunk:ta:o365:log" "Data input was interrupted by an unhandled exception." "File \"/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/portal.py\", line 31, in __init__" "TypeError: string indices must be integers"

I'm also sending a suggestion to MS to add events in the logs for "secret is expiring in X days".

[gordo32]

I also meant to mention that I'm using the same AppID for collecting AzureAD & Graph SecurityAPI logs.

Both of those continued to work for at least 10 days after O365 collector stopped working (SURPRISE!!).

No idea why these continued to work, even after *rebooting* my Heavy Forwarder.