Data Rotation Configuration of indexes.conf

[AdsicSplunk]

The problem I am facing is that my data is going from hot/warm bucket to frozen bucket directly. However, I want it to go to cold bucket first and then to frozen bucket. How can I configure the below configuration for my indexes.conf file to make the data go from hot to cold and then to frozen bucket. Please guide me.

coldPath = /colddb
homePath = /db
thawedPath = /thaweddb
coldToFrozenDir = /frozendb
maxHotBuckets = 3
maxWarmDBCount = 5
homepath.maxDataSizeMB = 5
maxHotSpanSecs = 180
maxTotalDataSizeMB = 20
frozenTimePeriodInSecs = 10800

[Elsurion · Accepted solution]

Hello,

The parameter frozenTimePeriodInSecs is the "bad" value for you problem, raise it from 3hrs to let's say 30days, then the data should be searchable in you cold folder and not already frozen.

frozenTimePeriodInSecs = 2592000

I have here even for large input of about 500Mio Events/day the online time set to 90days.

[AdsicSplunk]

Thank you for your inputs!!

I updated the above value mentioned for frozenTimePeriodInSecs = 2592000, but still it is going into frozen bucket before going in cold bucket. These values are kept low for understanding the actual bucket rotation practically. I need to see the data rotation happening in real-time by checking the size of the files and folders. In this case, cold bucket does not have any data file and it is going directly to frozen folder.

[Elsurion]

That can have now four reasons

1. You haven't restarted Splunk>
2. The size of the Data is more then 20MB
3. The Event itself is older then the 30days
4. The Timestamp of the event is wrongly interpreted as too old

To the 2 is said, if your Index is more the 20MB in the Warm Buckets the cold will not be considered. And the Events will be frozen when reaching the 20MB Limit.
http://docs.splunk.com/Documentation/Splunk/7.0.1/Indexer/HowSplunkstoresindexes

[AdsicSplunk]

The four reasons mentioned may be valid. Please find my comments below:-

1. You haven't restarted Splunk> I have restarted Splunk. And as I said I am just understanding the data movement, I need to see the data moving from hot/warm to cold but its going directly to frozen.

2. The size of the Data is more then 20MB - No it is not more than 20 MB

3. The Event itself is older then the 30days - No the events are real-time and I have removed the data and started to test with fresh one.

4. The Timestamp of the event is wrongly interpreted as too old- Timestamps are fine and live for any event that occurs.

I have a lost of storage, I am just using 20 MB to test this scenario of moving data from hot/warm to cold. Could you please help me in driving the values which will definitely work for checking this movement?

[Elsurion]

Ok, try this setup. I've browsed trough the definitions and crosschecked with my normal setup script.

maxHotBuckets = 3
maxWarmDBCount = 5
maxHotSpanSecs = 180
frozenTimePeriodInSecs = 2592000
maxWarmDBCount = 5
maxDataSize = auto
maxTotalDataSizeMB = 20
homePath.maxDataSizeMB = 0
coldPath.maxDataSizeMB = 0

I nulled the max home/cold IDX size, just to be sure we don't run there in a problem.

[AdsicSplunk]

I think the problem was with below attributes:-

homePath.maxDataSizeMB = 0
coldPath.maxDataSizeMB = 0

I could not find the exact reason but I think things are working for me now. If you could state a reason for its failure, I would really appreciate that.

Thank you for your help. I am marking the answer as accepted now. 🙂

[Elsurion]

According to the Doku, you can define there a global/indexbased value for your home and cold path.
I assume now (according this case), that both values have to be not null to work.
Since i'm normally not using it i cannot prove here, i took a note to test it on my own environment when i find the time ;).

https://docs.splunk.com/Documentation/Splunk/7.0.1/Indexer/Configureindexstoragesize

[AdsicSplunk]

Cool...
If you try this case in future, do not forget to comment in this post. However, if I get the answer to it, I will post the same. Thanks Elsurion. Cheers!!

[AdsicSplunk]

Hi Elsurion,

Now that I tested the rotation of event data, as checked today, I am seeing that the hot/warm buckets are always and it has stopped moving the data to cold bucket. The data again is moving to frozen bucket. Need your help again.

[AdsicSplunk]

Dont mind. I restarted Splunk and it started working.

[Elsurion]

This kind of solution I like the most 😉

[ddrillic]

Right.

This entire section is a bit off with very low values -

maxHotBuckets = 3
maxWarmDBCount = 5
homepath.maxDataSizeMB = 5
maxHotSpanSecs = 180
maxTotalDataSizeMB = 20
frozenTimePeriodInSecs = 10800

[Elsurion]

Depends on the input, my weather station has now 750MB with 14,8Mevt, but i'm collecting now sind 1 1/2 years.
But one part is the 3hrs delay for the freezing.