topic Re: Indexing data on Cold Bucket (log source based) in Getting Data In

Indexing data on Cold Bucket (log source based)

cemiam — Thu, 06 Jul 2017 13:53:22 GMT

Hi,

We are considering to index some of our data directly on cold buckets. They will not search frequently and we need to keep this data due to the regulations. As the log size is a bit large we have considered directly indexing this data on cold buckets. I have found many workarounds for data size and time periods but not able to find for log source based. Is there any workaround for this?

Best Regards,
Cem

Re: Indexing data on Cold Bucket (log source based)

woodcock — Thu, 06 Jul 2017 15:48:06 GMT

You don't reindex cold buckets, you thaw them:
https://docs.splunk.com/Documentation/Splunk/6.6.1/Indexer/Restorearchiveddata

Re: Indexing data on Cold Bucket (log source based)

cemiam — Thu, 06 Jul 2017 16:49:25 GMT

I think I am not able to reindex frozen buckets. I need to thaw frozen buckets to make them searchable. The cold buckets should be already seachable and not need to be thawed.

Re: Indexing data on Cold Bucket (log source based)

ddrillic — Mon, 10 Jul 2017 00:49:28 GMT

-- ... but not able to find for log source based

If you look at indexes.conf.spec

and at the ..default/indexes.conf, you can see that all stanzas are based on the index name. Sorry ; -)

Re: Indexing data on Cold Bucket (log source based)

esix_splunk — Mon, 10 Jul 2017 01:47:32 GMT

You dont need to reindex frozen. They are already indexed, all you have to do is move them to thaweddb path, and they will be searchable once you restart the Splunk instance.

You need to understand the bucket flow in Splunk a bit more I think, check this document : https://wiki.splunk.com/Deploy:BucketRotationAndRetention

Your buckets go through this lifecycle : HOT -> Warm -> COLD -> Frozen/Deleted, and THAWED if needed. In this, HOT/WARM are the same volume, whereas COLD can be on a different mount point, this is usually cheaper and slower storage.

Frozen is deleted by default, you'll need to script a move from COLD to Frozen if you plan to keep indexed data. If you have this, and need to search that data again, then you can move from the frozen media to the thawedDb path.

So moving directly from HOT to COLD isnt possible. You will need to tune the maxwarmDbCount so that Splunk wont retain the data in warm, and moves it directly to COLD after its rolled from HOT to WARM.

Re: Indexing data on Cold Bucket (log source based)

inventsekar — Mon, 10 Jul 2017 03:42:38 GMT

As said by Esix, i think, indexing data directly to COLD isnt possible.

make a custom index for this task.
set up HOT and WARM buckets with very low values that it will look like "almost" directly indexing to cold buckets.

Re: Indexing data on Cold Bucket (log source based)

cemiam — Mon, 10 Jul 2017 06:41:53 GMT

Thanks for the response. I will consider that. It seems like giving low amount of warm disk space will solve our problem.

Re: Indexing data on Cold Bucket (log source based)

cemiam — Mon, 10 Jul 2017 06:45:23 GMT

Thanks for the response. This will solve our issue 🙂