When the hot bucket is created on clustered envioemnet. What steps are followed.

[sat94541]

When the hot bucket is created on clustered envioemnet. What steps are followed.

[rbal_splunk]

When an indexer creates a hot bucket, it follows this procedure:

1) Indexer needs to create a new hot bucket
2) Indexer asks the CM whom to replicate the new hot bucket to
3) CM receives the request, checks the configured RF/SF, and selects indexers (randomly) as "targets" that the original indexer should stream to. For example, if RF=3, SF=2, it will respond with two targets, one of which will also be searchable (to satisfy SF=2).

All hot bucket replications are raw data ONLY. For the example above, the one target that is also told to be searchable will create its own tsidx files based on the rawdata that comes in.

[rbal_splunk]

Yes, we can survive failures, i.e. we won't stop indexing, but we cannot possibly meet replication policy for those source indexers that have a now failed peer in their target list. CM recovery and fixup needed asap, in that case
In other words: We really need CM HA.

[sat94541]

Does splunk say that ..... if your CM is down....we can survive any additional failures

[rbal_splunk]

correct.in which case new buckets dont meet rf/sf

[sat94541]

Until one of them fails as well, in which case.....?
The cluster cannot remain healthy until the CM comes back up...?

[sat94541]

Until one of them fails as well, in which case.....?
The cluster cannot remain healthy until the CM comes back up...?

[sat94541]

so with RF 3, it's going to be the same two indexers over and over

[rbal_splunk]

each indexer remembers the LAST list of targets the CM gave it. if the CM is down, it will continuously use the same targets for new hot buckets.
to be more correct, each indexer remembers the "last response of a new hot bucket request", and reuses that response

[sat94541]

we've believe that if the CM is down, we don't have to really break our necks in bringing it back up, e.g. an hour or more will be OK; in this instance, does the indexer always use the same peers to replicate to, or does it have a list of them it can use?