Bucket number exceeds maxHotBuckets value

[osec2a]

Hi,

In my splunk configurtation I have defined the maxHotBuckets to default value, so 3.
When I monitor my indexers I see that the number of Hot buckets exceeds the value 3.
Can anyone explain me why the number of hot buckets can exceed the maximum number of hot buckets?

Thanks

[osec2a]

@mmodestino This is my configuration in moniroting console

When I use dbinspect command to display each nomber of bucket per indexer this is what I get :

How can indexer have more than 3 (maxHotBuckets) hot buckets?

Thanks

[mattymo]

i recommend validating from the CLI. there is not enough info from dbinspect in the search above and I am not clear on whether you are even running a cluster?? I see 1 hot per indexer in that screenshot...Jump on the cli of that indexer and confirm what you are seeing in $SPLUNK_HOME/var/lib/splunk/<yourIndex>/db and look for the naming convention kellewic alluded to above. (hot_ and ####_GUID)

example from my standalone instance

[root@n00bserver db]# pwd
/home/splunker/splunk/var/lib/splunk/n00blab/db
[root@n00bserver db]# ls -la | grep hot
drwx--x---   3 splunker splunker  4096 Aug  9 15:02 hot_v1_342
drwx--x---   3 splunker splunker  4096 Aug  9 15:02 hot_v1_343

[kellewic]

dbinspect is counting the replicated hot buckets.

Look on one of your indexers for that index; you should see 3x buckets like "hot_" and N more like "####_GUID". Those can even exceed what dbinspect says but the ones with more than just "rawdata" is what's being counted in addition to the originating hot buckets.

Or if you have multiple pipelines, this can happen as well as inventsekar pointed out.

Easy check on the replicated front; try:

|dbinspect index=INDEX
|where state="hot"
|eval replicated=if(match(path, "/rb_"), "Y", if(state="hot" AND match(path, "/\d+_[-A-F0-9]+"), "Y", "N"))
|stats count by splunk_server, replicated
|sort replicated

Those with "N" will equal 3; those with "Y" will be the remainder.

Can also check:

|rest /services/data/indexes-extended
|where title="INDEX"
|table splunk_server, maxHotBuckets, bucket_dirs.home.hot_bucket_count

[inventsekar]

• Note that this limit will be applied per ingestion pipeline. For more information about multiple ingestion pipelines see parallelIngestionPipelines in server.conf.spec file.
• With N parallel ingestion pipelines the maximum number of hot buckets across all of the ingestion pipelines will be N * maxHotBuckets but maxHotBuckets for each ingestion pipeline. Each ingestion pipeline will independently write to and manage up to maxHotBuckets number of hot buckets. As a consequence of this, when multiple ingestion pipelines are used, there may be multiple (dependent on number of ingestion pipelines configured) hot buckets with events with overlapping time ranges.

https://docs.splunk.com/Documentation/Splunk/6.6.2/Admin/Indexesconf

[osec2a]

In server.conf parallelIngestionPipelines value is 1, so the maximum number of hot bucket is always 3.

[kellewic]

How are you monitoring them? Is it counting replicated hot buckets possibly?

[osec2a]

I have a distributed envrironment, so I manage all my indexers on a deploployer server
I don't think it is counting replicated hot buckets.
When I use dbinspect command, I see the same result

[mattymo]

can you be more specific and provide an example of what you are seeing?

so you are not clustering? And you are seeing more than 3 hot buckets per index? How many do you see??

[osec2a]

I add an answer beacause I can't add img in comments