Can I schedule rolling warmdb to colddb?

[richnavis]

Recently, I discovered that although the Splunk documentation indicates that colddb can be on slower storage, doing so has a performance impact on normal indexing processes because of the need to roll data from warm to colddbs. In my environment, I was able to reduce index blocking by putting colddbs on faster disk. My question, then is can rolling from warm to cold be done on a schedule, say, nightly, instead of dynamically. So, ideally, instead of constantly rolling data to cold, we would roll the oldest FULL Days data to cold, in a first in/first out manner.

Possible? Anyone doing this?

[kristian_kolb]

Well, you could do it like this (and I'm not saying that this best practice, or even good at all);

• Set your bucket size to something so large that the hot buckets don't roll to warm during the daytime.

• Have a cronjob for restarting splunkd at midnight (or whenever it suits you).

• The restart turn the hot buckets into warm, and the oldest warm gets rolled to cold.

Not sure this is helpful at all,

/Kristian

[kristian_kolb]

Not really that I know of, apart from the backup issue. There is probably a reason why auto high-volume is set to 10 GB and not more, but it might not be purely related to performance degradation of larger buckets. Then again, I'm just guessing here.

/k

[richnavis]

interesting idea... any downside to having large HOT buckets that you know of?

[araitz]

There is not a supported way of doing this, but you could use a cron script and Splunk's REST API to trigger a bucket roll. Some customers do this so that they have buckets that are for the most part 24 hours each, which can make backup and restore easier.

I wouldn't recommend doing this though - the small gain you get in performance, if any, will not outgain the risk and managment complexity involved in orchestrating this process.

[kristian_kolb]

Agree with you there 🙂

/k

[araitz]

Moving colddb to local seems like the solution to performance issues, not orchestrating bucket rolls 🙂

[richnavis]

WE did find that there is quite alot of blocking in our current setup (1500/day)... moving colddb local improved this to just a few 5-10 blocks per day.. so there IS quite a gain in performance to be had by addressing this problem..