How do I fix this issue? I found this search query that pulled up the indexes that was the problem.
Root Cause(s):
The percentage of small buckets (75%) created over the last hour is high and exceeded the red thresholds (50%) for index=_internal, and possibly more indexes, on this indexer. At the time this alert fired, total buckets created=4, small buckets=3
Query:
index=_internal sourcetype=splunkd component=HotBucketRoller "finished moving hot to warm"
| eval bucketSizeMB = round(size / 1024 / 1024, 2)
| table _time splunk_server idx bid bucketSizeMB
| rename idx as index
| join type=left index
[ | rest /services/data/indexes count=0
| rename title as index
| eval maxDataSize = case (maxDataSize == "auto", 750,
maxDataSize == "auto_high_volume", 10000,
true(), maxDataSize)
| table index updated currentDBSizeMB homePath.maxDataSizeMB maxDataSize maxHotBuckets maxWarmDBCount ]
| eval bucketSizePercent = round(100*(bucketSizeMB/maxDataSize))
| eval isSmallBucket = if (bucketSizePercent < 10, 1, 0)
| stats sum(isSmallBucket) as num_small_buckets
count as num_total_buckets
by index splunk_server
| eval percentSmallBuckets = round(100*(num_small_buckets/num_total_buckets))
| sort - percentSmallBuckets
| eval isViolation = if (percentSmallBuckets > 30, "Yes", "No")
After that I was able to see that main, metrics, and internal were in violation. But from there I am not sure how to determine which source type is causing the issue or how to fix it.
index=main
| eval latency=_indextime-_time
| stats min(latency),
max(latency),
avg(latency),
median(latency)
by index sourcetype
The following command is for when you have determined which sourcetype is causing the issue:
index=abc sourcetype=def
| eval latency=_indextime-_time
| stats min(latency),
max(latency),
avg(latency),
median(latency)
by index sourcetype host
were you able to fix this?