Splunk index db directory has extremely large number (over 30,000) of hot_v1_xxx directories. What's going on?

[jafaruddinlie]

Hi all
We are having issues with our Splunk install (performance slowly degrading over time) so I had a quick look at the logs.
It is Splunk 5.0.2 running on RHEL 6,

Turns out that under summarydb/db directory, I am seeing a lot of hot_v1_xxxxx directories (well, about 32000 of it) and SplunkD cannot create any more directories under it.
It looks like these directories are empty, is it safe to remove them?

[jrodman · Accepted solution]

It is safe, when splunk is stopped, to delete empty buckets in an index. (Either hot_v1_xxx or nonhot db_.... dirs). Empty index buckets can't help you for sure. (Caveat: In a cluster I'm less sure if "just delete it" is always the right action, maybe we will try to replicate the empty bucket if you delete it only one location.)

An empty hot could exist validly at the time between its creation and the first write to the location, but typically this is measured in fractions of a second. (Edge cases it might be measured in seconds, during strange deadlock bugs etc potentially minutes).

However there is a ceiling on the maximum hot buckets per index, so thousands of hot buckets at once is an invalid state for sure (unless this ceiling was altered? May want to review the btool output for splunk btool indexes list your_index) .

I suspect something is going wrong and may go wrong again, so you may want to poke through the errors and warnings in splunkd.log and possibly open a support case with a diag. http://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/Generateadiag

[jafaruddinlie]

Thanks 🙂
That is helpful, I have removed the empty buckets.
I'll keep an eye out if the folders are generated again.
That didn't help with the performance issue, another issue that I am still trying to get to the bottom of.