remove excess buckets

[jariw]

Hi,

some questions...

Last weekend we've got an error on the indexers. It is a multisite indexers with 6<>6 indexers (each site 6 indexers). 

Some indexers went down and the data storage went sky high. Stil not sure why. But when we started the indexers which where down, the data storage went partly back on nomal except one indexer.

I noticed a lot off excess buckets...  very very much. I started removing these buckets,  but it stopped on one point and never went further with cleaning.  Could this be because of the data part of this one indexer is full (it is at this moment in automatic detention state).

I don't see the activity on the cluster master, so it seems it is finished.. but i can't start a new action to clean, is says "Previously scheduled Remove Excess Buckets is running". 

I tried a rolling restart (in maint mode), but it doesn't allow because of the "remove excess buckets is running"..

How can i stop this "Previously scheduled Remove Excess Buckets is running"  ?

thanks in advance..

[pellegrini]

Restart splunkd on CM worked fine for me when "Previously scheduled Remove Excess Buckets is running", been hanging for days.

[jariw]

It solved itself after restart two ther indexers and remove excess buckets per index (what strangly was allowed).

After a while it seems there was enough space for removing the rest off the excess buckets.

[isoutamo]

Usually you could fix this kind of situations by rebooting CM.