Monitoring Splunk

Why is indexer receiving high small bucket creation warning?

mike_k
Path Finder

I am running a single instance Splunk Enterprise deployment (v. 8.1.3).

On the main GUI dashboard, I am getting a Red Health Status of Splunkd flag. On closer inspection, further detail is showing as Index Processor>Buckets with root cause "The percentage of small buckets (71%) created over the last hour is high and exceeded the red thresholds (50%) for index=os, and possibly more indexes, on this indexer. At the time this alert fired, total buckets created=11, small buckets=0"

What i can't quite figure out is, it is calling this a small bucket alert and yet the number of small buckets created=0.

I came across the following search online to do some further checking on this:

index=_internal sourcetype=splunkd component=HotBucketRoller "finished moving hot to warm"
 | eval bucketSizeMB = round(size / 1024 / 1024, 2)
 | table _time splunk_server idx bid bucketSizeMB
 | rename idx as index
 | join type=left index
     [ | rest /services/data/indexes count=0
       | rename title as index
       | eval maxDataSize = case (maxDataSize == "auto",             750,
                                  maxDataSize == "auto_high_volume", 10000,
                                  true(),                            maxDataSize)
       | table  index updated currentDBSizeMB homePath.maxDataSizeMB maxDataSize maxHotBuckets maxWarmDBCount ]
 | eval bucketSizePercent = round(100*(bucketSizeMB/maxDataSize))
 | eval isSmallBucket     = if (bucketSizePercent < 10, 1, 0)
 | stats sum(isSmallBucket) as num_small_buckets
         count              as num_total_buckets
         by index splunk_server
 | eval  percentSmallBuckets = round(100*(num_small_buckets/num_total_buckets))
 | sort  - percentSmallBuckets
 | eval isViolation = if (percentSmallBuckets > 30, "Yes", "No")

A Search over the last 24 hours is showing 4 buckets created (and no small buckets)

A search over the last 7 days is showing:

  • index="os", total buckets=10, number of small buckets=1
  • index="_internal", total buckets=38, number of small buckets=1

I guess i am a little intrigued as to why I am seeing this alert as i have had 2 small buckets created in the last week (and the percentage small buckets per index is at worst 10%).

Are there any other health checks that i should be looking at on my Indexer?

Labels (2)
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi @mike_k,

I had the same problem and I opened a case to Splunk Support.

The answer was that's a bug that will be probably solved in a next version of Splunk Enterprise, maybe 8.2.5.

Anyway, there isn't any relevant problem for the system.

Ciao.

Giuseppe

View solution in original post

gcusello
SplunkTrust
SplunkTrust

Hi @mike_k,

I had the same problem and I opened a case to Splunk Support.

The answer was that's a bug that will be probably solved in a next version of Splunk Enterprise, maybe 8.2.5.

Anyway, there isn't any relevant problem for the system.

Ciao.

Giuseppe

computermathguy
Explorer

We just upgraded to 8.2.6 and the bucket alerts still persist.  

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...