All Apps and Add-ons

AWS Add-On Data stopped ingesting - bucket error?


We are using the add-on, and it was working fine until suddenly, one of the cloudtrail inputs was no longer logging data. splunkd.log gives us this:

02-06-2019 16:25:55.018 +0000 WARN TcpOutputProc - Pipeline data does not have indexKey. [_path] = /opt/splunk/etc/apps/Splunk_TA_aws/bin/\n[_raw] = \n[_meta] = punct::\n[_stmid] = fJyw3RpcQ9LtT7\n[MetaData:Source] = source::aws_cloudwatch\n[MetaData:Host] = host::REDACTED\n[MetaData:Sourcetype] = sourcetype::aws_cloudwatch\n[_done] = _done\n[_linebreaker] = _linebreaker\n[_conf] = source::aws_cloudwatch|host::REDACTED|aws_cloudwatch|\n

The index does exist, and has data, it just isn't writing anything new. We have another cloudtrail input, going to a different index, that works fine.

This is probably related:

02-06-2019 16:22:26.214 +0000 INFO IndexWriter - Creating hot bucket=hot_v1_27, idx=aws_REDACTED, event timestamp=1549466074, reason="suitable bucket not found, number of hot buckets=0, max=3"
02-06-2019 16:22:26.215 +0000 INFO DatabaseDirectoryManager - idx=aws_REDACTED Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/aws_REDACTED/db', pendingBucketUpdates=0 . Reason='Adding bucket, bid=aws_REDACTED~27~8090F922-ECEC-4FF0-A4D8-99F6464206E1'

Tags (1)
0 Karma

Cloud trail was writing to an s3 bucket that just kept filling up and was never cleaned up. We wrote an s3 lifecycle policy you delete things older than x days. The aws ta was scanning the entire bucket and just couldn’t keep up once the bucket got over a certain size.
0 Karma



We are facing same issue with our app. Did you find any solution?

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In the last month, the Splunk Threat Research Team (STRT) has had 2 releases of new security content via the ...

Announcing the 1st Round Champion’s Tribute Winners of the Great Resilience Quest

We are happy to announce the 20 lucky questers who are selected to be the first round of Champion's Tribute ...

We’ve Got Education Validation!

Are you feeling it? All the career-boosting benefits of up-skilling with Splunk? It’s not just a feeling, it's ...