AWS S3 path-based access model deprecation, impact on cloudtrail data ingest and possible fixes

[soumyasaha25]

AWS had announced that they would deprecate deprecate the path-based access model that is used to specify the address of an object in an S3 bucket and this kicks in from 30th Sept 2020.
Example:
Current format:
https://s3.amazonaws.com/jbarr-public/images/ritchie_and_thompson_pdp11.jpeg
New format
https://jbarr-public.s3.amazonaws.com/images/ritchie_and_thompson_pdp11.jpeg
More info on this can be found here

My question is, what changes needs to be done on splunk configs end so that we continue to receive data (in my case cloudtrail) from S3 buckets with the new naming convention.
i can only see from the config files that the only place where the "bucket_name " and "hostname is referanced is in inputs.conf in splunk_TA_aws.
Do i need an upgrade of the Splunk TA to support this. i am currently on Splunk TA version 4.5.0 and splunk version 7.1.1

[amiracle]

For starters, you should upgrade your Splunk instance to 8.x, the AWS App to 6.0 and the Add-on to 5.0 since they will use the newer Boto SDK v3 along with Python 3.7.

I don't believe that this will have an impact on how Splunk collects the data from the S3 buckets since we are using the Both SDK to pull the bucket names and not the URL's. Regardless of what the URL is, the bucket name will stay the same and that is what is being used for data collection.

If you want you can see what Splunk sees by going to the AWS cli and typing in aws s3 ls and you should only see a list of your S3 buckets.

[amiracle]

Just as a side note, if you want to collect CloudTrail data at scale, you might want to look into using Grand Central : https://www.splunk.com/en_us/blog/partners/trumpeting-to-grand-central-monitor-and-deploy-cloud-base...