All Apps and Add-ons

What should I do about the following Office 365 AD audit log errors: "TypeError: 'int' object is not iterable"

crisponions2
Explorer

I am having an issue with the Splunk add on for Office 365. It has been working somewhat fine for a couple months and then yesterday I started getting these errors.

2018-09-19 15:32:15,292 level=ERROR pid=28491 tid=MainThread logger=splunk_ta_o365.modinputs.management_activity pos=utils.py:wrapper:67 | start_time=1537396334 datainput="mgmt_ad_audit" | message="Data input was interrupted by an unhandled exception." 
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunksdc/utils.py", line 65, in wrapper
    return func(*args, **kwargs)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/management_activity.py", line 88, in run
    with app.open_checkpoint(self.name) as checkpoint:
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunksdc/collector.py", line 258, in open_checkpoint
    checkpoint = LocalKVStore.open_always(fullname)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunksdc/checkpoint.py", line 167, in open_always
    indexes = cls.build_indexes(fp)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunksdc/checkpoint.py", line 174, in build_indexes
    for flag, key, pos in cls._replay(fp):
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunksdc/checkpoint.py", line 103, in _replay
    flag, key, _ = umsgpack.unpack(fp)
TypeError: 'int' object is not iterable

I am still receiving the general_audit logs without issue.

Any ideas?

jaxjohnny2000
Builder

deleting the checkpoint files did not work for me.

2018-12-18 18:12:20,645 level=ERROR pid=77680 tid=MainThread logger=splunk_ta_o365.modinputs.management_activity pos=utils.py:wrapper:67 | start_time=1545156724 datainput="management_activity_audit_azure_ad" | message="Data input was interrupted by an unhandled exception."
Traceback (most recent call last):
File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunksdc/utils.py", line 65, in wrapper
return func(*args, **kwargs)
File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/management_activity.py", line 91, in run
executor.run(adapter)
File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunksdc/batch.py", line 62, in run
delegate.done(job, result)
File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/management_activity.py", line 148, in done
self._ingest_content_blob(content, result)
File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/management_activity.py", line 167, in _ingest_content_blob
self._event_writer.write_fileobj(data, source=content.uri)
File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunksdc/event_writer.py", line 160, in write_fileobj
self._write(data)
File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunksdc/event_writer.py", line 132, in _write
self._dev.write(data)
IOError: [Errno 32] Broken pipe

0 Karma

orca
Explorer

Hey @jaxjohnny2000 Did you find any solution to this Broken Pipe error?

0 Karma

Lowell
Super Champion

I strongly recommend avoiding the Splunk Add-on for Microsoft Office 365 at this time (especially v1.0.0). Use the O365 inputs for the Splunk Add-on for Microsoft Cloud Services instead and save yourself a ton of pain. I really wish Splunk would properly acknowledge how broken this app is and stop recommending it over MSCS which actually works, I suspect it will be several more releases before they get it right which at this point could be years away! But please file a bug and report because the more complains equals more pressure to get this fixed.

Because event if deleting all your checkpoint files really did work, that's a really sucky "fix".

0 Karma

Lowell
Super Champion

I think the issue you experienced is that the checkpoint file for your particular input was corrupted. That's why some of your inputs would continue to work while one stopped. I just ran into this myself, though the Python exception was different (Unable to decode utf-8) the source of the problem is likely the same.

The modular inputs track their state in checkpoint files (.ckpt), which live in a path something like this: $SPLUNK_HOME/var/lib/splunk/modinputs/<input-type>/<input_name>.ckpt And if they get corrupted, they need to be cleaned up. Otherwise, the modular input doesn't know what data it's previously ingested and what data is "new". Unfortunately, the input just gives up in case where the state is corrupted.

For example, in my case, the file that was corrupt was named:

/opt/splunk/var/lib/splunk/modinputs/splunk_ta_o365_management_activity/Management_Activity_AzureActiveDirectory.ckpt

At this point the easy option is to simply delete the file and the modulare input will start over. Depending on how far back the specific input will look by default (often a few days) that may be acceptable. The other option is to attempt to repair the file.

I wrote a simple little script to help accomplish this. It's not very polished but it works in a pinch. Some usage notes are included in the comments at the top. Hopefully this helps anyone who runs into this issue later.

https://gist.github.com/lowell80/93f31c6275a908ef94f048f8ad8149d2

0 Karma

crisponions2
Explorer

I ended up deleting and reinstalling the add on, pointing to my existing indexes and now everything is normal again.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...