Deployment Architecture

Why does the number of temporary buckets exceed the number of replicated buckets in an indexer cluster?

immortalraghava
Path Finder

Hi All,

I am following this document to get an understanding about the indexer cluster bucket replication in Splunk.
http://docs.splunk.com/Documentation/Splunk/6.0/Indexer/HowSplunkstoresindexes#Warm.2Fcold_bucket_na...

Excerpt
Note: In an indexer cluster, when data is streaming from the originating peer to a target peer, the data first goes into a temporary directory on the target peer, which is identified by the originating peer's &ltlocalid&gt and &ltguid&gt, like this: &ltlocalid&gt_&ltguid&gt. This is true, independent of the type of bucket the data is being streamed from. When the replication has completed, the directory is rolled into a warm bucket, identified by the rb_ prefix, as described above.

Also parallely I had a demo indexer cluster system for my testing. Interestingly during my tests I found that the temporary buckets (localid_guid) are in larger number than the actual rb_buckets (replicated buckets).

What could be the possible explanation for this ?

Any help appreciated thanks!

0 Karma
1 Solution

sowings
Splunk Employee
Splunk Employee

The temporary buckets you describe are searchable hot buckets. They're just actively open for writing "now". You'll wait for a few possibilities: the bucket going idle (maxHotIdleSecs), the bucket being crowded out (maxHotBuckets), reaching its largest size (maxDataSize, default "auto" or 750MB unless your index is "main", in which case it's "auto_high_volume" which is 10 GB on a 64-bit platform). Also, a restart (on the "start" part of the stop/start) will force buckets to roll from hot to warm. There are a few edge cases in an indexing cluster related to a loss of connectivity with the cluster master as well.

In any event, just wait, and eventually you'll have many more rb than the hot searchable copies!

View solution in original post

sowings
Splunk Employee
Splunk Employee

The temporary buckets you describe are searchable hot buckets. They're just actively open for writing "now". You'll wait for a few possibilities: the bucket going idle (maxHotIdleSecs), the bucket being crowded out (maxHotBuckets), reaching its largest size (maxDataSize, default "auto" or 750MB unless your index is "main", in which case it's "auto_high_volume" which is 10 GB on a 64-bit platform). Also, a restart (on the "start" part of the stop/start) will force buckets to roll from hot to warm. There are a few edge cases in an indexing cluster related to a loss of connectivity with the cluster master as well.

In any event, just wait, and eventually you'll have many more rb than the hot searchable copies!

immortalraghava
Path Finder

Is there a specific property which controls the size of the temporary bucket before it renames to replicated bucket?

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...