Indexing data on Cold Bucket (log source based)

[cemiam]

Hi,

We are considering to index some of our data directly on cold buckets. They will not search frequently and we need to keep this data due to the regulations. As the log size is a bit large we have considered directly indexing this data on cold buckets. I have found many workarounds for data size and time periods but not able to find for log source based. Is there any workaround for this?

Best Regards,
Cem

[inventsekar · Accepted solution]

As said by Esix, i think, indexing data directly to COLD isnt possible.

make a custom index for this task.
set up HOT and WARM buckets with very low values that it will look like "almost" directly indexing to cold buckets.

[cemiam]

Thanks for the response. This will solve our issue 🙂

[ddrillic]

-- ... but not able to find for log source based

If you look at indexes.conf.spec

and at the ..default/indexes.conf, you can see that all stanzas are based on the index name. Sorry ; -)

[woodcock]

You don't reindex cold buckets, you thaw them:
https://docs.splunk.com/Documentation/Splunk/6.6.1/Indexer/Restorearchiveddata

[cemiam]

I think I am not able to reindex frozen buckets. I need to thaw frozen buckets to make them searchable. The cold buckets should be already seachable and not need to be thawed.

[esix_splunk]

You dont need to reindex frozen. They are already indexed, all you have to do is move them to thaweddb path, and they will be searchable once you restart the Splunk instance.

You need to understand the bucket flow in Splunk a bit more I think, check this document : https://wiki.splunk.com/Deploy:BucketRotationAndRetention

Your buckets go through this lifecycle : HOT -> Warm -> COLD -> Frozen/Deleted, and THAWED if needed. In this, HOT/WARM are the same volume, whereas COLD can be on a different mount point, this is usually cheaper and slower storage.

Frozen is deleted by default, you'll need to script a move from COLD to Frozen if you plan to keep indexed data. If you have this, and need to search that data again, then you can move from the frozen media to the thawedDb path.

So moving directly from HOT to COLD isnt possible. You will need to tune the maxwarmDbCount so that Splunk wont retain the data in warm, and moves it directly to COLD after its rolled from HOT to WARM.

[cemiam]

Thanks for the response. I will consider that. It seems like giving low amount of warm disk space will solve our problem.