Getting Data In

Indexing data on Cold Bucket (log source based)

cemiam
Path Finder

Hi,

We are considering to index some of our data directly on cold buckets. They will not search frequently and we need to keep this data due to the regulations. As the log size is a bit large we have considered directly indexing this data on cold buckets. I have found many workarounds for data size and time periods but not able to find for log source based. Is there any workaround for this?

Best Regards,
Cem

Tags (1)
0 Karma
1 Solution

inventsekar
SplunkTrust
SplunkTrust

As said by Esix, i think, indexing data directly to COLD isnt possible.

make a custom index for this task.
set up HOT and WARM buckets with very low values that it will look like "almost" directly indexing to cold buckets.

thanks and best regards,
Sekar

PS - If this or any post helped you in any way, pls consider upvoting, thanks for reading !

View solution in original post

0 Karma

inventsekar
SplunkTrust
SplunkTrust

As said by Esix, i think, indexing data directly to COLD isnt possible.

make a custom index for this task.
set up HOT and WARM buckets with very low values that it will look like "almost" directly indexing to cold buckets.

thanks and best regards,
Sekar

PS - If this or any post helped you in any way, pls consider upvoting, thanks for reading !
0 Karma

cemiam
Path Finder

Thanks for the response. This will solve our issue 🙂

0 Karma

ddrillic
Ultra Champion

-- ... but not able to find for log source based

If you look at indexes.conf.spec

and at the ..default/indexes.conf, you can see that all stanzas are based on the index name. Sorry ; -)

0 Karma

woodcock
Esteemed Legend
0 Karma

cemiam
Path Finder

I think I am not able to reindex frozen buckets. I need to thaw frozen buckets to make them searchable. The cold buckets should be already seachable and not need to be thawed.

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

You dont need to reindex frozen. They are already indexed, all you have to do is move them to thaweddb path, and they will be searchable once you restart the Splunk instance.

You need to understand the bucket flow in Splunk a bit more I think, check this document : https://wiki.splunk.com/Deploy:BucketRotationAndRetention

Your buckets go through this lifecycle : HOT -> Warm -> COLD -> Frozen/Deleted, and THAWED if needed. In this, HOT/WARM are the same volume, whereas COLD can be on a different mount point, this is usually cheaper and slower storage.

Frozen is deleted by default, you'll need to script a move from COLD to Frozen if you plan to keep indexed data. If you have this, and need to search that data again, then you can move from the frozen media to the thawedDb path.

So moving directly from HOT to COLD isnt possible. You will need to tune the maxwarmDbCount so that Splunk wont retain the data in warm, and moves it directly to COLD after its rolled from HOT to WARM.

cemiam
Path Finder

Thanks for the response. I will consider that. It seems like giving low amount of warm disk space will solve our problem.

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In November, the Splunk Threat Research Team had one release of new security content via the Enterprise ...

Index This | Divide 100 by half. What do you get?

November 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Stay Connected: Your Guide to December Tech Talks, Office Hours, and Webinars!

❄️ Celebrate the season with our December lineup of Community Office Hours, Tech Talks, and Webinars! ...