All Apps and Add-ons

What should I do about the following Office 365 AD audit log errors: "TypeError: 'int' object is not iterable"

crisponions2
Explorer

I am having an issue with the Splunk add on for Office 365. It has been working somewhat fine for a couple months and then yesterday I started getting these errors.

2018-09-19 15:32:15,292 level=ERROR pid=28491 tid=MainThread logger=splunk_ta_o365.modinputs.management_activity pos=utils.py:wrapper:67 | start_time=1537396334 datainput="mgmt_ad_audit" | message="Data input was interrupted by an unhandled exception." 
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunksdc/utils.py", line 65, in wrapper
    return func(*args, **kwargs)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/management_activity.py", line 88, in run
    with app.open_checkpoint(self.name) as checkpoint:
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunksdc/collector.py", line 258, in open_checkpoint
    checkpoint = LocalKVStore.open_always(fullname)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunksdc/checkpoint.py", line 167, in open_always
    indexes = cls.build_indexes(fp)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunksdc/checkpoint.py", line 174, in build_indexes
    for flag, key, pos in cls._replay(fp):
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunksdc/checkpoint.py", line 103, in _replay
    flag, key, _ = umsgpack.unpack(fp)
TypeError: 'int' object is not iterable

I am still receiving the general_audit logs without issue.

Any ideas?

jaxjohnny2000
Builder

deleting the checkpoint files did not work for me.

2018-12-18 18:12:20,645 level=ERROR pid=77680 tid=MainThread logger=splunk_ta_o365.modinputs.management_activity pos=utils.py:wrapper:67 | start_time=1545156724 datainput="management_activity_audit_azure_ad" | message="Data input was interrupted by an unhandled exception."
Traceback (most recent call last):
File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunksdc/utils.py", line 65, in wrapper
return func(*args, **kwargs)
File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/management_activity.py", line 91, in run
executor.run(adapter)
File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunksdc/batch.py", line 62, in run
delegate.done(job, result)
File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/management_activity.py", line 148, in done
self._ingest_content_blob(content, result)
File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/management_activity.py", line 167, in _ingest_content_blob
self._event_writer.write_fileobj(data, source=content.uri)
File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunksdc/event_writer.py", line 160, in write_fileobj
self._write(data)
File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunksdc/event_writer.py", line 132, in _write
self._dev.write(data)
IOError: [Errno 32] Broken pipe

0 Karma

Lowell
Super Champion

I strongly recommend avoiding the Splunk Add-on for Microsoft Office 365 at this time (especially v1.0.0). Use the O365 inputs for the Splunk Add-on for Microsoft Cloud Services instead and save yourself a ton of pain. I really wish Splunk would properly acknowledge how broken this app is and stop recommending it over MSCS which actually works, I suspect it will be several more releases before they get it right which at this point could be years away! But please file a bug and report because the more complains equals more pressure to get this fixed.

Because event if deleting all your checkpoint files really did work, that's a really sucky "fix".

0 Karma

Lowell
Super Champion

I think the issue you experienced is that the checkpoint file for your particular input was corrupted. That's why some of your inputs would continue to work while one stopped. I just ran into this myself, though the Python exception was different (Unable to decode utf-8) the source of the problem is likely the same.

The modular inputs track their state in checkpoint files (.ckpt), which live in a path something like this: $SPLUNK_HOME/var/lib/splunk/modinputs/<input-type>/<input_name>.ckpt And if they get corrupted, they need to be cleaned up. Otherwise, the modular input doesn't know what data it's previously ingested and what data is "new". Unfortunately, the input just gives up in case where the state is corrupted.

For example, in my case, the file that was corrupt was named:

/opt/splunk/var/lib/splunk/modinputs/splunk_ta_o365_management_activity/Management_Activity_AzureActiveDirectory.ckpt

At this point the easy option is to simply delete the file and the modulare input will start over. Depending on how far back the specific input will look by default (often a few days) that may be acceptable. The other option is to attempt to repair the file.

I wrote a simple little script to help accomplish this. It's not very polished but it works in a pinch. Some usage notes are included in the comments at the top. Hopefully this helps anyone who runs into this issue later.

https://gist.github.com/lowell80/93f31c6275a908ef94f048f8ad8149d2

0 Karma

crisponions2
Explorer

I ended up deleting and reinstalling the add on, pointing to my existing indexes and now everything is normal again.

0 Karma
Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!