topic Re: Moving Splunk data to S3 bucket in a Cluster environment in Getting Data In

Moving Splunk data to S3 bucket in a Cluster environment

shrogers — Wed, 20 Jan 2021 19:53:21 GMT

Hi Everyone,

I'm looking for a working package that can move data from the Splunk cluster environment to the S3 bucket for archiving. All examples I'm getting does work.

Re: Moving Splunk data to S3 bucket in a Cluster environment

richgalloway — Wed, 20 Jan 2021 20:12:43 GMT

Well, there's SmartStore, which is built-in to Splunk.

Beyond that, we'll need more information. Is your Splunk on-prem, private cloud, or Splunk Cloud? How do you want the data stored in S3 (searchable by Splunk or something else)?

What examples have you tried so far?

Re: Moving Splunk data to S3 bucket in a Cluster environment

shrogers — Wed, 20 Jan 2021 21:57:21 GMT

Thank you your quick response. Smartstore would require a whole new setup and we are not able to go down that route.

It's an on-prem cluster environment. We just want to archive index data to S3 after 90 days. If we need to get it searchable, we'll get it done manually.

Re: Moving Splunk data to S3 bucket in a Cluster environment

richgalloway — Thu, 21 Jan 2021 13:21:53 GMT

Write a coldToFrozenScript. This script is invoked by Splunk when a bucket is due to be archived. See 'coldToFrozenScript ' in the Admin manual (https://docs.splunk.com/Documentation/Splunk/8.1.1/Admin/Indexesconf#indexes.conf.spec)