All Apps and Add-ons

Amazon Web Services Add-on: s3 generic error - TypeError: 'int' object is not iterable

kristian_kolb
Ultra Champion

Following a server crash (unknown reason) the collection of S3 logs has partially broken down. Seeing the following in _internal (after enabling DEBUG logging for the S3 generic part).

Any ideas on how to get it working again?

2018-09-03 14:38:22,122 level=INFO pid=15104 tid=Thread-4 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:_do_index_data:104 
  | datainput="securitylogs_blahblah_staging" bucket_name="blahblah-staging-ext-securitylogs-20180215145qqqqqqq00000001", job_uid="757azzzz-xxxx-4a5b-b735-yyyyy31754e" start_time=1535978302 
  | message="Start processing."


2018-09-03 14:38:22,123 level=DEBUG pid=15104 tid=Thread-4 logger=splunksdc.checkpoint pos=checkpoint.py:build_indexes:169 
  | datainput="securitylogs_blahblah_staging" bucket_name="blahblah-staging-ext-securitylogs-20180215145qqqqqqq00000001", job_uid="757azzzz-xxxx-4a5b-b735-yyyyye31754e" start_time=1535978302 
  | message="Key was set." pos=221 key="securitylogs/ls.s3.2cf5f2c6-xxxx-4947-bb01-yyyyy96c7e22.2018-08-01T22.30.part11921.txt.gz"

some more of the above DEBUG message - different key, but same Job_uid, and then it ends with:

2018-09-03 14:38:22,123 level=ERROR pid=15104 tid=Thread-4 logger=splunk_ta_aws.modinputs.generic_s3.aws_s3_data_loader pos=aws_s3_data_loader.py:index_data:90 
   | datainput="securitylogs_blahblah_staging" bucket_name="blahblah-staging-ext-securitylogs-20180215145qqqqqqq00000001" 
   | message="Failed to collect data through generic S3." job_uid="757azzz-xxxx-4a5b-b735-yyyyy31754e" start_time=1535978302 Traceback (most recent call last): 
   File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_data_loader.py", line 85, in index_data self._do_index_data() 
   File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_data_loader.py", line 106, in _do_index_data self.collect_data() 
   File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_data_loader.py", line 139, in collect_data index_store = s3ckpt.S3IndexCheckpointer(self._config) 
   File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_checkpointer.py", line 141, in __init__ config[asc.data_input], config[tac.checkpoint_dir] 
   File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_checkpointer.py", line 80, in get S3CkptPool.S3CkptItem(ckpt_name, ckpt_dir) 
   File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunk_ta_aws/modinputs/generic_s3/aws_s3_checkpointer.py", line 65, in __init__ self.idx_ckpt = LocalKVStore.open_always(ckpt_file_idx) 
   File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunksdc/checkpoint.py", line 156, in open_always indexes = cls.build_indexes(fp) 
   File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunksdc/checkpoint.py", line 163, in build_indexes for flag, key, pos in cls._replay(fp): 
   File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/splunksdc/checkpoint.py", line 92, in _replay flag, key, _ = umsgpack.unpack(fp) TypeError: 'int' object is not iterable

Something seems to have gotten corrupted. Have tried to disable/enable the inputs from within the AWS-addon-app-GUI, but with little success.

pkiripolsky
Path Finder

Hey there kristian,
Not sure if you're still having this issue but I was able to resolve a very similar situation recently on my own environment.

I had the same errors that you were seeing in the log stream which pointed me to the modinputs for this add on. Here's what I did:

  1. Add-on was installed on our search head so I stopped the search head
  2. Then, I went into /opt/splunk/var/lib/modinputs and deleted the following directories (rm -rf)
    • aws_* (anything that started with aws basically)
    • splunk_ta_aws_sqs
  3. I restarted the search head and disabled/re-enabled the aws inputs within the add-on. Shortly thereafter, I started getting events in again.

I'd also like to point out that prior to going through these steps I had disabled and re-enabled the add-on and restarted my search head as well as re-entered in the input credentials but none of those helped me.

I'm still not certain why this behavior occurred in the first place and why it broke in this particular way but this was the solution that helped me out. Cheers!

phudinhha
Explorer

Hi everyone,

 

I followed this suggestion and got my problem solved. However, there is one thing that i would like to highlight is that, after did this, the whole index is gone, and i have to rebuild my index.

0 Karma

pkiripolsky
Path Finder

Apologies on that @phudinhha  -- I usually separate out my indexes from downloadedTA/Apps and create my own app to contain all indexes so it didn't occur to me that this may happen. However, if your indexed data is being sent to separate indexers (i.e. your search head isn't an all in one), then the index is only gone from the search head but the data should still be on the indexer(s). You ought to be able to recreate the index on the search head and that will allow you to search against your indexed data on the indexers again. 

 

0 Karma

kristian_kolb
Ultra Champion

Sorry for the late reply @pkiripolsky . Got it working, but I can't recall exactly how. Could have been your suggested solution. Thanks.

benjarvissainsb
Engager

Thanks for the help pkiripolsky, it helped me to resolve the same issue just now.

Get Updates on the Splunk Community!

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...