In my splunk configurtation I have defined the maxHotBuckets to default value, so 3.
When I monitor my indexers I see that the number of Hot buckets exceeds the value 3.
Can anyone explain me why the number of hot buckets can exceed the maximum number of hot buckets?
I have a distributed envrironment, so I manage all my indexers on a deploployer server
I don't think it is counting replicated hot buckets.
When I use dbinspect command, I see the same result
can you be more specific and provide an example of what you are seeing?
so you are not clustering? And you are seeing more than 3 hot buckets per index? How many do you see??
dbinspect is counting the replicated hot buckets.
Look on one of your indexers for that index; you should see 3x buckets like "hot" and N more like "####GUID". Those can even exceed what dbinspect says but the ones with more than just "rawdata" is what's being counted in addition to the originating hot buckets.
Or if you have multiple pipelines, this can happen as well as inventsekar pointed out.
Easy check on the replicated front; try:
|dbinspect index=INDEX |where state="hot" |eval replicated=if(match(path, "/rb_"), "Y", if(state="hot" AND match(path, "/\d+_[-A-F0-9]+"), "Y", "N")) |stats count by splunk_server, replicated |sort replicated
Those with "N" will equal 3; those with "Y" will be the remainder.
Can also check:
|rest /services/data/indexes-extended |where title="INDEX" |table splunk_server, maxHotBuckets, bucket_dirs.home.hot_bucket_count
i recommend validating from the CLI. there is not enough info from dbinspect in the search above and I am not clear on whether you are even running a cluster?? I see 1 hot per indexer in that screenshot...Jump on the cli of that indexer and confirm what you are seeing in
$SPLUNK_HOME/var/lib/splunk/<yourIndex>/db and look for the naming convention kellewic alluded to above. (hot_ and ####_GUID)
example from my standalone instance
[root@n00bserver db]# pwd /home/splunker/splunk/var/lib/splunk/n00blab/db [root@n00bserver db]# ls -la | grep hot drwx--x--- 3 splunker splunker 4096 Aug 9 15:02 hot_v1_342 drwx--x--- 3 splunker splunker 4096 Aug 9 15:02 hot_v1_343