All Apps and Add-ons

How to connect s3 buckets in Glacier to the Splunk App for AWS?

Explorer

We are having problems with s3 bucket injection. Our corporate security policy states we need to keep 2 years of our ELb logs in s3. So we lifecycle them into glacier. This unfortunately means that when we try to connect to them with the AWS app, we get a whole lot of files with a different storage type and this causes thousands of errors. For some reason, this kills the process so we can't get new data. When I look at the source, I do see the number going up, but we can't seem to get anything newer than the first time this ran which was about a week ago.

These are the sanitized errors we see constantly scrolling through aws_s3.log:

2015-10-20 22:55:12,240 ERROR pid=22473 tid=MainThread file=aws_s3.py:stream_events:868 | Incomplete: bucket: 'OURBUCKET' key: u'OURPREFIX/AWSLogs/OURACCOUNT#/elasticloadbalancing/OURZONE/2014/05/19/OURACCOUNT#_elasticloadbalancing_OURZONE_OURELBNAME_20140519T0100Z_10.241.4.64_3indsmej.log' etag: "2464d51635ffc8954e36b59f479f72cc" attempt_number: 2 orig_size: 1119509 bytes_streamed: 0 total_bytes_streamed: 0 Exception: S3ResponseError: 403 Forbidden - InvalidObjectState - The operation is not valid for the object's storage class
0 Karma

Splunk Employee
Splunk Employee

Is there any data in s3 bucket "OURBUCKET"? I think that's just empty bucket for listing purpose only. If so, you can just skip that folder either use whitelist or folder name filter in config page.

0 Karma