Monitoring Splunk

Why is indexer receiving high small bucket creation warning?

mike_k
Path Finder

I am running a single instance Splunk Enterprise deployment (v. 8.1.3).

On the main GUI dashboard, I am getting a Red Health Status of Splunkd flag. On closer inspection, further detail is showing as Index Processor>Buckets with root cause "The percentage of small buckets (71%) created over the last hour is high and exceeded the red thresholds (50%) for index=os, and possibly more indexes, on this indexer. At the time this alert fired, total buckets created=11, small buckets=0"

What i can't quite figure out is, it is calling this a small bucket alert and yet the number of small buckets created=0.

I came across the following search online to do some further checking on this:

index=_internal sourcetype=splunkd component=HotBucketRoller "finished moving hot to warm"
 | eval bucketSizeMB = round(size / 1024 / 1024, 2)
 | table _time splunk_server idx bid bucketSizeMB
 | rename idx as index
 | join type=left index
     [ | rest /services/data/indexes count=0
       | rename title as index
       | eval maxDataSize = case (maxDataSize == "auto",             750,
                                  maxDataSize == "auto_high_volume", 10000,
                                  true(),                            maxDataSize)
       | table  index updated currentDBSizeMB homePath.maxDataSizeMB maxDataSize maxHotBuckets maxWarmDBCount ]
 | eval bucketSizePercent = round(100*(bucketSizeMB/maxDataSize))
 | eval isSmallBucket     = if (bucketSizePercent < 10, 1, 0)
 | stats sum(isSmallBucket) as num_small_buckets
         count              as num_total_buckets
         by index splunk_server
 | eval  percentSmallBuckets = round(100*(num_small_buckets/num_total_buckets))
 | sort  - percentSmallBuckets
 | eval isViolation = if (percentSmallBuckets > 30, "Yes", "No")

A Search over the last 24 hours is showing 4 buckets created (and no small buckets)

A search over the last 7 days is showing:

  • index="os", total buckets=10, number of small buckets=1
  • index="_internal", total buckets=38, number of small buckets=1

I guess i am a little intrigued as to why I am seeing this alert as i have had 2 small buckets created in the last week (and the percentage small buckets per index is at worst 10%).

Are there any other health checks that i should be looking at on my Indexer?

Labels (2)
0 Karma
1 Solution

gcusello
Esteemed Legend

Hi @mike_k,

I had the same problem and I opened a case to Splunk Support.

The answer was that's a bug that will be probably solved in a next version of Splunk Enterprise, maybe 8.2.5.

Anyway, there isn't any relevant problem for the system.

Ciao.

Giuseppe

View solution in original post

gcusello
Esteemed Legend

Hi @mike_k,

I had the same problem and I opened a case to Splunk Support.

The answer was that's a bug that will be probably solved in a next version of Splunk Enterprise, maybe 8.2.5.

Anyway, there isn't any relevant problem for the system.

Ciao.

Giuseppe

Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) v3.54.0

The Splunk Threat Research Team (STRT) recently released Enterprise Security Content Update (ESCU) v3.54.0 and ...

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

New Learning Videos on Topics Most Requested by You! Plus This Month’s New Splunk ...

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...