Can we integrate AWS S3 with our premise Splunk installation for non-hot data?

[asherlohman]

We are running Splunk on premise using bare metal servers with local disk. The amount of disk space locally is limited so we can only now store a limited amount of data in terms of time. We would like to expand the amount of data we are indexing over time so we can look at historical trends, etc. Has anyone ever used AWS S3 (or similar) to store data beyond the "hot" tier (i.e. medium, color, frozen)?

[kiyototamura]

One approach is to use open source log management tools like Fluentd so that you can unify and simplify your logging layer. Using the output plugins for Splunk and Amazon S3, you can send same data to both Splunk and S3. You can learn more about Fluentd and its enterprise offering below:

https://www.fluentd.org
https://fluentd.treasuredata.com
https://fluentd.treasuredata.com/splunk-optimize/

[awurster]

@asherlohman there's a few other answers i've commented on which should help regarding frozen -> s3.

https://answers.splunk.com/answers/56522/frozen-archives-into-amazon-s3.html
https://answers.splunk.com/answers/293894/how-to-put-cold-and-frozen-data-on-s3-in-aws.html