Deployment Architecture

When the hot bucket is created on clustered envioemnet. What steps are followed.

Communicator

When the hot bucket is created on clustered envioemnet. What steps are followed.

0 Karma

Splunk Employee
Splunk Employee

When an indexer creates a hot bucket, it follows this procedure:

1) Indexer needs to create a new hot bucket
2) Indexer asks the CM whom to replicate the new hot bucket to
3) CM receives the request, checks the configured RF/SF, and selects indexers (randomly) as "targets" that the original indexer should stream to. For example, if RF=3, SF=2, it will respond with two targets, one of which will also be searchable (to satisfy SF=2).

All hot bucket replications are raw data ONLY. For the example above, the one target that is also told to be searchable will create its own tsidx files based on the rawdata that comes in.

Splunk Employee
Splunk Employee

Yes, we can survive failures, i.e. we won't stop indexing, but we cannot possibly meet replication policy for those source indexers that have a now failed peer in their target list. CM recovery and fixup needed asap, in that case
In other words: We really need CM HA.

0 Karma

Communicator

Does splunk say that ..... if your CM is down....we can survive any additional failures

0 Karma

Splunk Employee
Splunk Employee

correct.in which case new buckets dont meet rf/sf

0 Karma

Communicator

Until one of them fails as well, in which case.....?
The cluster cannot remain healthy until the CM comes back up...?

0 Karma

Communicator

Until one of them fails as well, in which case.....?
The cluster cannot remain healthy until the CM comes back up...?

0 Karma

Communicator

so with RF 3, it's going to be the same two indexers over and over

0 Karma

Splunk Employee
Splunk Employee

each indexer remembers the LAST list of targets the CM gave it. if the CM is down, it will continuously use the same targets for new hot buckets.
to be more correct, each indexer remembers the "last response of a new hot bucket request", and reuses that response

0 Karma

Communicator

we've believe that if the CM is down, we don't have to really break our necks in bringing it back up, e.g. an hour or more will be OK; in this instance, does the indexer always use the same peers to replicate to, or does it have a list of them it can use?

0 Karma