Splunk Enterprise

How to configure secondary storage device as Cold Destination? and move the data from hot bucket to secondary storage(cold bucket)???

swati_sharma
New Member

How to configure secondary storage device as Cold Destination? and move the data from hot bucket to secondary storage(cold bucket)???

Tags (2)
0 Karma

rsennett_splunk
Splunk Employee
Splunk Employee

You can easily move it from Hot to Warm by either restarting Splunk (all hot rolls to warm) or changing the following entry in indexes.conf for the stanza matching this index :
maxHotBucket=1

To move the data from Warm to Cold you'll need to reduce the size of the index so all buckets will roll from Warm to Cold.

So... find the indexes.conf definition
edit the coldPath
temporarily change (or add) this value maxWarmDBCount=1
Everything will roll to cold if there is data still coming in. You could probably set both of them to 0 (although I've never tried it) but that seems like a runaway train...

The point is, you're using the settings on your index to force the data to roll out of hot and warm and into cold. This is of course if you still have data flowing in.

Keep in mind, as I said at the top, when you stop Splunk, all hot buckets will roll to warm. So if there is only one warm bucket, and data still flowing in... everything ends up in cold... quickly.

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!
0 Karma

swati_sharma
New Member

Thank you for your guidance, Can you please tell me the parameters which I need to change to get the data in cold bucket from hot bucket???

0 Karma

rsennett_splunk
Splunk Employee
Splunk Employee

swati_sharma: I changed the answer so it reflects the correct directive.
Basically, you are forcing the data to pass thru by reducing the buckets. Assuming you have data still flowing through, setting the 'bucket size' in both hot and warm to 1 will cause the data to quickly flow to cold.

All of this is documented as mentioned above.

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!
0 Karma

swati_sharma
New Member

I have tried with the given settings by you i.e maxHotBucket=1 and maxTotalDataSizeMB=0, However still I am not getting in the cold bucket, The behaviour is data directly move to the frozen bucket form hot bucket.

0 Karma

rsennett_splunk
Splunk Employee
Splunk Employee

You're right. I've edited my answer... check out the indexes.conf doc.
http://docs.splunk.com/Documentation/Splunk/6.2.4/admin/Indexesconf
the size directive governed the entire index... so put that back where it was. (mea culpa)
if you reduce the number of hot buckets, and then also the number of warm buckets, your stuff will have nowhere to go but cold.

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!
0 Karma
Get Updates on the Splunk Community!

Introducing Splunk Enterprise 9.2

WATCH HERE! Watch this Tech Talk to learn about the latest features and enhancements shipped in the new Splunk ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...