Hello, I'm not finding info on the limits within Splunk's data rebalancing. Some context, I have ~40 indexers and stood up 8 new ones. The 40 old ones had an avg of ~150k buckets each. At some point the rebalance reported that it was completed (above the .9 threshold) even though there were only ~40k buckets on the new indexers. When I kicked off a second rebalance, it started from 20% again and continued rebalancing because the new indexers were NOT space limited on the smartstore caches yet. The timeout was set to 11 hours and the first one finished in ~4. The master did not restart during this balancing. Can anyone shed some more light on why the first rebalance died? Like, is there a 350k bucket limit per rebalance or something? ... View more

Hello, I am experiencing a periodic issue with smartstore where a bucket will try to be evicted then proceeds to fail and does that cycle thousands of times. The indexer IO is fine, the bucket is warm, we have enough cache sizing, and I have not been able to correlate any cache logs with when these failures begin on multiple indexer nodes in the cluster (~33% of indexers). 2 questions: * What is an urgent mode eviction? * What can cause warm buckets to be unable to be evicted when they rolled to warm ~a full day earlier? ... View more