Monitoring Splunk

Why is indexer receiving high small bucket creation warning?

mike_k
Path Finder

I am running a single instance Splunk Enterprise deployment (v. 8.1.3).

On the main GUI dashboard, I am getting a Red Health Status of Splunkd flag. On closer inspection, further detail is showing as Index Processor>Buckets with root cause "The percentage of small buckets (71%) created over the last hour is high and exceeded the red thresholds (50%) for index=os, and possibly more indexes, on this indexer. At the time this alert fired, total buckets created=11, small buckets=0"

What i can't quite figure out is, it is calling this a small bucket alert and yet the number of small buckets created=0.

I came across the following search online to do some further checking on this:

index=_internal sourcetype=splunkd component=HotBucketRoller "finished moving hot to warm"
 | eval bucketSizeMB = round(size / 1024 / 1024, 2)
 | table _time splunk_server idx bid bucketSizeMB
 | rename idx as index
 | join type=left index
     [ | rest /services/data/indexes count=0
       | rename title as index
       | eval maxDataSize = case (maxDataSize == "auto",             750,
                                  maxDataSize == "auto_high_volume", 10000,
                                  true(),                            maxDataSize)
       | table  index updated currentDBSizeMB homePath.maxDataSizeMB maxDataSize maxHotBuckets maxWarmDBCount ]
 | eval bucketSizePercent = round(100*(bucketSizeMB/maxDataSize))
 | eval isSmallBucket     = if (bucketSizePercent < 10, 1, 0)
 | stats sum(isSmallBucket) as num_small_buckets
         count              as num_total_buckets
         by index splunk_server
 | eval  percentSmallBuckets = round(100*(num_small_buckets/num_total_buckets))
 | sort  - percentSmallBuckets
 | eval isViolation = if (percentSmallBuckets > 30, "Yes", "No")

A Search over the last 24 hours is showing 4 buckets created (and no small buckets)

A search over the last 7 days is showing:

  • index="os", total buckets=10, number of small buckets=1
  • index="_internal", total buckets=38, number of small buckets=1

I guess i am a little intrigued as to why I am seeing this alert as i have had 2 small buckets created in the last week (and the percentage small buckets per index is at worst 10%).

Are there any other health checks that i should be looking at on my Indexer?

Labels (2)
0 Karma
1 Solution

gcusello
Esteemed Legend

Hi @mike_k,

I had the same problem and I opened a case to Splunk Support.

The answer was that's a bug that will be probably solved in a next version of Splunk Enterprise, maybe 8.2.5.

Anyway, there isn't any relevant problem for the system.

Ciao.

Giuseppe

View solution in original post

gcusello
Esteemed Legend

Hi @mike_k,

I had the same problem and I opened a case to Splunk Support.

The answer was that's a bug that will be probably solved in a next version of Splunk Enterprise, maybe 8.2.5.

Anyway, there isn't any relevant problem for the system.

Ciao.

Giuseppe

computermathguy
Engager

We just upgraded to 8.2.6 and the bucket alerts still persist.  

0 Karma
Get Updates on the Splunk Community!

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...

Security Highlights | January 2023 Newsletter

January 2023 Splunk Security Essentials (SSE) 3.7.0 ReleaseThe free Splunk Security Essentials (SSE) 3.7.0 app ...