The default for rotation from warm to cold is 300. I am retaining about 1 years worth of data in all indexes and most of that data is kept in warm buckets I have about 13.22 TB of "homepath" data and 9.08 TB of "coldpath" data. If I change the default for warm to cold rotation from 300 to 150 I will move about 6.5 TB into cold storage. this will allow me to put the cold buckets on slower SAN space.
My question is,
What will happen when I start up splunk with this new rotation policy?
Will splunk 6.3 chock when trying to move 6.5 TB of data from a fast SAN to a Slower SAN?
I have been asked to do this as a cost saving to the service.
When you restart your splunk instance, Splunk should start rolling older Warm buckets into Cold bucket, keeping only 150 (latest) Warm buckets in warm bucket directory.
I would not say it would choke but it would take some time and will show high CPU usage based on the amount of data and write speed of your slower SAN for cold bucket storage.
When you restart your splunk instance, Splunk should start rolling older Warm buckets into Cold bucket, keeping only 150 (latest) Warm buckets in warm bucket directory.
I would not say it would choke but it would take some time and will show high CPU usage based on the amount of data and write speed of your slower SAN for cold bucket storage.
Another suggestion would be to do it in 2-3 steps, change it to 250, then 200 and then final 150. Check its performance after changing it to 250, if it doesn't affect searching and indexing , then you can directly reduce it to 150 from 250.
Thanks @Somesoni2 you always have a wise and guiding response. I will try the phased approach you suggested.