All Apps and Add-ons

AWS S3 path-based access model deprecation, impact on cloudtrail data ingest and possible fixes


AWS had announced that they would deprecate deprecate the path-based access model that is used to specify the address of an object in an S3 bucket and this kicks in from 30th Sept 2020.
Current format:
New format
More info on this can be found here

My question is, what changes needs to be done on splunk configs end so that we continue to receive data (in my case cloudtrail) from S3 buckets with the new naming convention.
i can only see from the config files that the only place where the "bucket_name " and "hostname is referanced is in inputs.conf in splunk_TA_aws.
Do i need an upgrade of the Splunk TA to support this. i am currently on Splunk TA version 4.5.0 and splunk version 7.1.1

0 Karma

Splunk Employee
Splunk Employee

For starters, you should upgrade your Splunk instance to 8.x, the AWS App to 6.0 and the Add-on to 5.0 since they will use the newer Boto SDK v3 along with Python 3.7.

I don't believe that this will have an impact on how Splunk collects the data from the S3 buckets since we are using the Both SDK to pull the bucket names and not the URL's. Regardless of what the URL is, the bucket name will stay the same and that is what is being used for data collection.

If you want you can see what Splunk sees by going to the AWS cli and typing in aws s3 ls and you should only see a list of your S3 buckets.

0 Karma

Splunk Employee
Splunk Employee

Just as a side note, if you want to collect CloudTrail data at scale, you might want to look into using Grand Central :

0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...