How can I read a public dataset from public s3 bucket?

[jdunlea1]

Is there a way (using the Splunk TA for AWS or otherwise) that Splunk can connect to a publicly available S3 bucket (such as those made available here https://registry.opendata.aws/) and read in the data?

From the Splunk TA, the only buckets that I can read from are those which were created in my account.

[jdunlea1 · Accepted solution]

After some further digging and testing it appears that it can be done but you need to create the input using the conf file as per link text

The key here for ingesting "old" data from a public dataset in S3 is that you need to set initial_scan_datetime to be a date that is BEFORE the modified file time for the files in the S3 bucket.

Once I did this, I was able to pull the public dataset into Splunk from the public S3 bucket.