AWS Add-On Data stopped ingesting - bucket error?

[jtm7x2]

We are using the add-on, and it was working fine until suddenly, one of the cloudtrail inputs was no longer logging data. splunkd.log gives us this:

02-06-2019 16:25:55.018 +0000 WARN TcpOutputProc - Pipeline data does not have indexKey. [_path] = /opt/splunk/etc/apps/Splunk_TA_aws/bin/aws_cloudwatch.py\n[_raw] = \n[_meta] = punct::\n[_stmid] = fJyw3RpcQ9LtT7\n[MetaData:Source] = source::aws_cloudwatch\n[MetaData:Host] = host::REDACTED\n[MetaData:Sourcetype] = sourcetype::aws_cloudwatch\n[_done] = _done\n[_linebreaker] = _linebreaker\n[_conf] = source::aws_cloudwatch|host::REDACTED|aws_cloudwatch|\n

The index does exist, and has data, it just isn't writing anything new. We have another cloudtrail input, going to a different index, that works fine.

This is probably related:

02-06-2019 16:22:26.214 +0000 INFO IndexWriter - Creating hot bucket=hot_v1_27, idx=aws_REDACTED, event timestamp=1549466074, reason="suitable bucket not found, number of hot buckets=0, max=3"
02-06-2019 16:22:26.215 +0000 INFO DatabaseDirectoryManager - idx=aws_REDACTED Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/aws_REDACTED/db', pendingBucketUpdates=0 . Reason='Adding bucket, bid=aws_REDACTED~27~8090F922-ECEC-4FF0-A4D8-99F6464206E1'

[jtm7x2]

Cloud trail was writing to an s3 bucket that just kept filling up and was never cleaned up. We wrote an s3 lifecycle policy you delete things older than x days. The aws ta was scanning the entire bucket and just couldn’t keep up once the bucket got over a certain size.

[priyanka_231019]

Hi,

We are facing same issue with our app. Did you find any solution?