topic Re: Can I schedule rolling warmdb to colddb? in Deployment Architecture

Can I schedule rolling warmdb to colddb?

richnavis — Fri, 11 May 2012 12:57:47 GMT

Recently, I discovered that although the Splunk documentation indicates that colddb can be on slower storage, doing so has a performance impact on normal indexing processes because of the need to roll data from warm to colddbs. In my environment, I was able to reduce index blocking by putting colddbs on faster disk. My question, then is can rolling from warm to cold be done on a schedule, say, nightly, instead of dynamically. So, ideally, instead of constantly rolling data to cold, we would roll the oldest FULL Days data to cold, in a first in/first out manner.

Possible? Anyone doing this?

Re: Can I schedule rolling warmdb to colddb?

araitz — Fri, 11 May 2012 13:51:47 GMT

There is not a supported way of doing this, but you could use a cron script and Splunk's REST API to trigger a bucket roll. Some customers do this so that they have buckets that are for the most part 24 hours each, which can make backup and restore easier.

I wouldn't recommend doing this though - the small gain you get in performance, if any, will not outgain the risk and managment complexity involved in orchestrating this process.

Re: Can I schedule rolling warmdb to colddb?

kristian_kolb — Fri, 11 May 2012 16:09:35 GMT

Well, you could do it like this (and I'm not saying that this best practice, or even good at all);

Set your bucket size to something so large that the hot buckets don't roll to warm during the daytime.
Have a cronjob for restarting splunkd at midnight (or whenever it suits you).
The restart turn the hot buckets into warm, and the oldest warm gets rolled to cold.

Not sure this is helpful at all,

/Kristian

Re: Can I schedule rolling warmdb to colddb?

richnavis — Fri, 11 May 2012 20:28:26 GMT

interesting idea... any downside to having large HOT buckets that you know of?

Re: Can I schedule rolling warmdb to colddb?

richnavis — Fri, 11 May 2012 20:39:25 GMT

WE did find that there is quite alot of blocking in our current setup (1500/day)... moving colddb local improved this to just a few 5-10 blocks per day.. so there IS quite a gain in performance to be had by addressing this problem..

Re: Can I schedule rolling warmdb to colddb?

araitz — Fri, 11 May 2012 20:42:25 GMT

Moving colddb to local seems like the solution to performance issues, not orchestrating bucket rolls 🙂

Re: Can I schedule rolling warmdb to colddb?

kristian_kolb — Fri, 11 May 2012 21:20:19 GMT

Agree with you there 🙂

Re: Can I schedule rolling warmdb to colddb?

kristian_kolb — Fri, 11 May 2012 21:23:53 GMT

Not really that I know of, apart from the backup issue. There is probably a reason why auto high-volume is set to 10 GB and not more, but it might not be purely related to performance degradation of larger buckets. Then again, I'm just guessing here.