All Apps and Add-ons

Amazon Web Services Add-on: s3 generic error - TypeError: 'int' object is not iterable

Ultra Champion

Following a server crash (unknown reason) the collection of S3 logs has partially broken down. Seeing the following in _internal (after enabling DEBUG logging for the S3 generic part).

Any ideas on how to get it working again?

2018-09-03 14:38:22,122 level=INFO pid=15104 tid=Thread-4 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader 
  | datainput="securitylogs_blahblah_staging" bucket_name="blahblah-staging-ext-securitylogs-20180215145qqqqqqq00000001", job_uid="757azzzz-xxxx-4a5b-b735-yyyyy31754e" start_time=1535978302 
  | message="Start processing."

2018-09-03 14:38:22,123 level=DEBUG pid=15104 tid=Thread-4 logger=splunksdc.checkpoint 
  | datainput="securitylogs_blahblah_staging" bucket_name="blahblah-staging-ext-securitylogs-20180215145qqqqqqq00000001", job_uid="757azzzz-xxxx-4a5b-b735-yyyyye31754e" start_time=1535978302 
  | message="Key was set." pos=221 key="securitylogs/ls.s3.2cf5f2c6-xxxx-4947-bb01-yyyyy96c7e22.2018-08-01T22.30.part11921.txt.gz"

some more of the above DEBUG message - different key, but same Job_uid, and then it ends with:

2018-09-03 14:38:22,123 level=ERROR pid=15104 tid=Thread-4 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader 
   | datainput="securitylogs_blahblah_staging" bucket_name="blahblah-staging-ext-securitylogs-20180215145qqqqqqq00000001" 
   | message="Failed to collect data through generic S3." job_uid="757azzz-xxxx-4a5b-b735-yyyyy31754e" start_time=1535978302 Traceback (most recent call last): 
   File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/", line 85, in index_data self._do_index_data() 
   File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/", line 106, in _do_index_data self.collect_data() 
   File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/", line 139, in collect_data index_store = s3ckpt.S3IndexCheckpointer(self._config) 
   File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/", line 141, in __init__ config[asc.data_input], config[tac.checkpoint_dir] 
   File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/", line 80, in get S3CkptPool.S3CkptItem(ckpt_name, ckpt_dir) 
   File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/", line 65, in __init__ self.idx_ckpt = LocalKVStore.open_always(ckpt_file_idx) 
   File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunksdc/", line 156, in open_always indexes = cls.build_indexes(fp) 
   File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunksdc/", line 163, in build_indexes for flag, key, pos in cls._replay(fp): 
   File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunksdc/", line 92, in _replay flag, key, _ = umsgpack.unpack(fp) TypeError: 'int' object is not iterable

Something seems to have gotten corrupted. Have tried to disable/enable the inputs from within the AWS-addon-app-GUI, but with little success.

Path Finder

Hey there kristian,
Not sure if you're still having this issue but I was able to resolve a very similar situation recently on my own environment.

I had the same errors that you were seeing in the log stream which pointed me to the modinputs for this add on. Here's what I did:

  1. Add-on was installed on our search head so I stopped the search head
  2. Then, I went into /opt/splunk/var/lib/modinputs and deleted the following directories (rm -rf)
    • aws_* (anything that started with aws basically)
    • splunk_ta_aws_sqs
  3. I restarted the search head and disabled/re-enabled the aws inputs within the add-on. Shortly thereafter, I started getting events in again.

I'd also like to point out that prior to going through these steps I had disabled and re-enabled the add-on and restarted my search head as well as re-entered in the input credentials but none of those helped me.

I'm still not certain why this behavior occurred in the first place and why it broke in this particular way but this was the solution that helped me out. Cheers!


Hi everyone,


I followed this suggestion and got my problem solved. However, there is one thing that i would like to highlight is that, after did this, the whole index is gone, and i have to rebuild my index.

0 Karma

Path Finder

Apologies on that @phudinhha  -- I usually separate out my indexes from downloadedTA/Apps and create my own app to contain all indexes so it didn't occur to me that this may happen. However, if your indexed data is being sent to separate indexers (i.e. your search head isn't an all in one), then the index is only gone from the search head but the data should still be on the indexer(s). You ought to be able to recreate the index on the search head and that will allow you to search against your indexed data on the indexers again. 


0 Karma

Ultra Champion

Sorry for the late reply @pkiripolsky . Got it working, but I can't recall exactly how. Could have been your suggested solution. Thanks.


Thanks for the help pkiripolsky, it helped me to resolve the same issue just now.

Get Updates on the Splunk Community!

Admin Your Splunk Cloud, Your Way

Join us to maximize different techniques to best tune Splunk Cloud. In this Tech Enablement, you will get ...

Cloud Platform | Discontinuing support for TLS version 1.0 and 1.1

Overview Transport Layer Security (TLS) is a security communications protocol that lets two computers, ...

New Customer Testimonials

Enterprises of all sizes and across different industries are accelerating cloud adoption by migrating ...