All Apps and Add-ons

Amazon Web Services Add-on: s3 generic error - TypeError: 'int' object is not iterable

Ultra Champion

Following a server crash (unknown reason) the collection of S3 logs has partially broken down. Seeing the following in _internal (after enabling DEBUG logging for the S3 generic part).

Any ideas on how to get it working again?

2018-09-03 14:38:22,122 level=INFO pid=15104 tid=Thread-4 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:104 
  | datainput="securitylogs_blahblah_staging" bucket_name="blahblah-staging-ext-securitylogs-20180215145qqqqqqq00000001", job_uid="757azzzz-xxxx-4a5b-b735-yyyyy31754e" start_time=1535978302 
  | message="Start processing."


2018-09-03 14:38:22,123 level=DEBUG pid=15104 tid=Thread-4 logger=splunksdc.checkpoint pos=checkpoint.py:build_indexes:169 
  | datainput="securitylogs_blahblah_staging" bucket_name="blahblah-staging-ext-securitylogs-20180215145qqqqqqq00000001", job_uid="757azzzz-xxxx-4a5b-b735-yyyyye31754e" start_time=1535978302 
  | message="Key was set." pos=221 key="securitylogs/ls.s3.2cf5f2c6-xxxx-4947-bb01-yyyyy96c7e22.2018-08-01T22.30.part11921.txt.gz"

some more of the above DEBUG message - different key, but same Job_uid, and then it ends with:

2018-09-03 14:38:22,123 level=ERROR pid=15104 tid=Thread-4 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:index_data:90 
   | datainput="securitylogs_blahblah_staging" bucket_name="blahblah-staging-ext-securitylogs-20180215145qqqqqqq00000001" 
   | message="Failed to collect data through generic S3." job_uid="757azzz-xxxx-4a5b-b735-yyyyy31754e" start_time=1535978302 Traceback (most recent call last): 
   File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_data_loader.py", line 85, in index_data self._do_index_data() 
   File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_data_loader.py", line 106, in _do_index_data self.collect_data() 
   File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_data_loader.py", line 139, in collect_data index_store = s3ckpt.S3IndexCheckpointer(self._config) 
   File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_checkpointer.py", line 141, in __init__ config[asc.data_input], config[tac.checkpoint_dir] 
   File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_checkpointer.py", line 80, in get S3CkptPool.S3CkptItem(ckpt_name, ckpt_dir) 
   File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_checkpointer.py", line 65, in __init__ self.idx_ckpt = LocalKVStore.open_always(ckpt_file_idx) 
   File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunksdc/checkpoint.py", line 156, in open_always indexes = cls.build_indexes(fp) 
   File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunksdc/checkpoint.py", line 163, in build_indexes for flag, key, pos in cls._replay(fp): 
   File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunksdc/checkpoint.py", line 92, in _replay flag, key, _ = umsgpack.unpack(fp) TypeError: 'int' object is not iterable

Something seems to have gotten corrupted. Have tried to disable/enable the inputs from within the AWS-addon-app-GUI, but with little success.

Explorer

Hey there kristian,
Not sure if you're still having this issue but I was able to resolve a very similar situation recently on my own environment.

I had the same errors that you were seeing in the log stream which pointed me to the modinputs for this add on. Here's what I did:

  1. Add-on was installed on our search head so I stopped the search head
  2. Then, I went into /opt/splunk/var/lib/modinputs and deleted the following directories (rm -rf)
    • aws_* (anything that started with aws basically)
    • splunk_ta_aws_sqs
  3. I restarted the search head and disabled/re-enabled the aws inputs within the add-on. Shortly thereafter, I started getting events in again.

I'd also like to point out that prior to going through these steps I had disabled and re-enabled the add-on and restarted my search head as well as re-entered in the input credentials but none of those helped me.

I'm still not certain why this behavior occurred in the first place and why it broke in this particular way but this was the solution that helped me out. Cheers!

Loves-to-Learn Lots

Hi everyone,

 

I followed this suggestion and got my problem solved. However, there is one thing that i would like to highlight is that, after did this, the whole index is gone, and i have to rebuild my index.

0 Karma

Ultra Champion

Sorry for the late reply @pkiripolsky . Got it working, but I can't recall exactly how. Could have been your suggested solution. Thanks.

0 Karma

Thanks for the help pkiripolsky, it helped me to resolve the same issue just now.

0 Karma