All Apps and Add-ons

AWS Add-On Data stopped ingesting - bucket error?

jtm7x2
Explorer

We are using the add-on, and it was working fine until suddenly, one of the cloudtrail inputs was no longer logging data. splunkd.log gives us this:

02-06-2019 16:25:55.018 +0000 WARN TcpOutputProc - Pipeline data does not have indexKey. [_path] = /opt/splunk/etc/apps/Splunk_TA_aws/bin/aws_cloudwatch.py\n[_raw] = \n[_meta] = punct::\n[_stmid] = fJyw3RpcQ9LtT7\n[MetaData:Source] = source::aws_cloudwatch\n[MetaData:Host] = host::REDACTED\n[MetaData:Sourcetype] = sourcetype::aws_cloudwatch\n[_done] = _done\n[_linebreaker] = _linebreaker\n[_conf] = source::aws_cloudwatch|host::REDACTED|aws_cloudwatch|\n

The index does exist, and has data, it just isn't writing anything new. We have another cloudtrail input, going to a different index, that works fine.

This is probably related:

02-06-2019 16:22:26.214 +0000 INFO IndexWriter - Creating hot bucket=hot_v1_27, idx=aws_REDACTED, event timestamp=1549466074, reason="suitable bucket not found, number of hot buckets=0, max=3"
02-06-2019 16:22:26.215 +0000 INFO DatabaseDirectoryManager - idx=aws_REDACTED Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/aws_REDACTED/db', pendingBucketUpdates=0 . Reason='Adding bucket, bid=aws_REDACTED~27~8090F922-ECEC-4FF0-A4D8-99F6464206E1'

Tags (1)
0 Karma

jtm7x2
Explorer
Cloud trail was writing to an s3 bucket that just kept filling up and was never cleaned up. We wrote an s3 lifecycle policy you delete things older than x days. The aws ta was scanning the entire bucket and just couldn’t keep up once the bucket got over a certain size.
0 Karma

priyanka_231019
Explorer

Hi,

We are facing same issue with our app. Did you find any solution?

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...