I am running a single instance Splunk Enterprise deployment (v. 8.1.3).
On the main GUI dashboard, I am getting a Red Health Status of Splunkd flag. On closer inspection, further detail is showing as Index Processor>Buckets with root cause "The percentage of small buckets (71%) created over the last hour is high and exceeded the red thresholds (50%) for index=os, and possibly more indexes, on this indexer. At the time this alert fired, total buckets created=11, small buckets=0"
What i can't quite figure out is, it is calling this a small bucket alert and yet the number of small buckets created=0.
I came across the following search online to do some further checking on this:
index=_internal sourcetype=splunkd component=HotBucketRoller "finished moving hot to warm"
| eval bucketSizeMB = round(size / 1024 / 1024, 2)
| table _time splunk_server idx bid bucketSizeMB
| rename idx as index
| join type=left index
[ | rest /services/data/indexes count=0
| rename title as index
| eval maxDataSize = case (maxDataSize == "auto", 750,
maxDataSize == "auto_high_volume", 10000,
true(), maxDataSize)
| table index updated currentDBSizeMB homePath.maxDataSizeMB maxDataSize maxHotBuckets maxWarmDBCount ]
| eval bucketSizePercent = round(100*(bucketSizeMB/maxDataSize))
| eval isSmallBucket = if (bucketSizePercent < 10, 1, 0)
| stats sum(isSmallBucket) as num_small_buckets
count as num_total_buckets
by index splunk_server
| eval percentSmallBuckets = round(100*(num_small_buckets/num_total_buckets))
| sort - percentSmallBuckets
| eval isViolation = if (percentSmallBuckets > 30, "Yes", "No")
A Search over the last 24 hours is showing 4 buckets created (and no small buckets)
A search over the last 7 days is showing:
- index="os", total buckets=10, number of small buckets=1
- index="_internal", total buckets=38, number of small buckets=1
I guess i am a little intrigued as to why I am seeing this alert as i have had 2 small buckets created in the last week (and the percentage small buckets per index is at worst 10%).
Are there any other health checks that i should be looking at on my Indexer?
