All Apps and Add-ons

AWS Add-On Data stopped ingesting - bucket error?


We are using the add-on, and it was working fine until suddenly, one of the cloudtrail inputs was no longer logging data. splunkd.log gives us this:

02-06-2019 16:25:55.018 +0000 WARN TcpOutputProc - Pipeline data does not have indexKey. [_path] = /opt/splunk/etc/apps/Splunk_TA_aws/bin/\n[_raw] = \n[_meta] = punct::\n[_stmid] = fJyw3RpcQ9LtT7\n[MetaData:Source] = source::aws_cloudwatch\n[MetaData:Host] = host::REDACTED\n[MetaData:Sourcetype] = sourcetype::aws_cloudwatch\n[_done] = _done\n[_linebreaker] = _linebreaker\n[_conf] = source::aws_cloudwatch|host::REDACTED|aws_cloudwatch|\n

The index does exist, and has data, it just isn't writing anything new. We have another cloudtrail input, going to a different index, that works fine.

This is probably related:

02-06-2019 16:22:26.214 +0000 INFO IndexWriter - Creating hot bucket=hot_v1_27, idx=aws_REDACTED, event timestamp=1549466074, reason="suitable bucket not found, number of hot buckets=0, max=3"
02-06-2019 16:22:26.215 +0000 INFO DatabaseDirectoryManager - idx=aws_REDACTED Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/aws_REDACTED/db', pendingBucketUpdates=0 . Reason='Adding bucket, bid=aws_REDACTED~27~8090F922-ECEC-4FF0-A4D8-99F6464206E1'

Tags (1)
0 Karma

Cloud trail was writing to an s3 bucket that just kept filling up and was never cleaned up. We wrote an s3 lifecycle policy you delete things older than x days. The aws ta was scanning the entire bucket and just couldn’t keep up once the bucket got over a certain size.
0 Karma



We are facing same issue with our app. Did you find any solution?

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Customer Survey!

If you use Splunk Observability Cloud, we invite you to share your valuable insights with us through a brief ...

.conf23 | Get Your Cybersecurity Defense Analyst Certification in Vegas

We’re excited to announce a new Splunk certification exam being released at .conf23! If you’re going to Las ...

Starting With Observability: OpenTelemetry Best Practices

Tech Talk Starting With Observability: OpenTelemetry Best Practices Tuesday, October 17, 2023   |  11AM PST / ...