All Apps and Add-ons

How to use shuttl to archive frozen data in local FS

alanwill
Explorer

I'm trying to implement shuttl in our day to day Splunk workflow and in setting it up, it looks like it would shuttl my cold data to the new frozen location, say S3. However I already have many terabytes of frozen data stored locally that I'd like to shuttl to S3, can I still use shuttl for this?

The idea would be to copy the existing local mounted SAN frozen archives to S3, then use the coldToFrozenScript from then on to move directly to S3.

How can I do this? Thanks,
alan

Tags (2)
0 Karma
1 Solution

Petter_Eriksson
Splunk Employee
Splunk Employee

You could setup Shuttl and call its copy/freeze scripts manually.

Here's how you do it:
1. Install and configure Shuttl.
2. Start Splunk and make sure Shuttl is running by looking at the Shuttl dashboard.
3. call $SPLUNK_HOME/etc/apps/shuttl/bin/warmToColdScript.sh /absolute/path/to/bucket /absolute/path/to/bucket
And your buckets should now be copied.

And yes, I've written "/absolute/path/to/bucket" twice. That's because usually when you use that script, you want to move the bucket from Splunk's warm to cold directory.

Alternative: Use this modified script, placed in Shuttl's bin directory: https://gist.github.com/petterik/7793825
Call it with a single argument of the bucket's absolute path. Installing, configuring and starting Shuttl is still needed.

View solution in original post

0 Karma

Petter_Eriksson
Splunk Employee
Splunk Employee

You could setup Shuttl and call its copy/freeze scripts manually.

Here's how you do it:
1. Install and configure Shuttl.
2. Start Splunk and make sure Shuttl is running by looking at the Shuttl dashboard.
3. call $SPLUNK_HOME/etc/apps/shuttl/bin/warmToColdScript.sh /absolute/path/to/bucket /absolute/path/to/bucket
And your buckets should now be copied.

And yes, I've written "/absolute/path/to/bucket" twice. That's because usually when you use that script, you want to move the bucket from Splunk's warm to cold directory.

Alternative: Use this modified script, placed in Shuttl's bin directory: https://gist.github.com/petterik/7793825
Call it with a single argument of the bucket's absolute path. Installing, configuring and starting Shuttl is still needed.

View solution in original post

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!