topic Re: indexer replication in Deployment Architecture

indexer replication

dkr3500 — Fri, 04 Jun 2021 21:53:58 GMT

I stood up a new set of indexers this weekend and set my RF=2 and SF=1 on the CM with the hope that the old indexers will replicate the data and populate the new set of indexers (I'm going to eventually decom the old indexers).

However, that doesn't seem to be working. All the peers are up and running, and UFs are sending data correctly via indexer_discovery...but the data isn't being replicated. My question is, why aren't my indexes (other than these native ones) showing up here on the CM? (Settings > Indexer clustering > Indexes tab:

Only native indexes are showing up on cluster master.

Re: indexer replication

s2_splunk — Fri, 04 Jun 2021 22:00:47 GMT

What did your deployment look like before, was it already a cluster, albeit a smaller one?

If you did not have a cluster before, your existing data will sit in standalone buckets and will not replicate by default. Also, when using clustering, all peer definitions (indexes.conf) is done on the CM (master-apps) and bundle-pushed to the peers. This is how the CM knows about indexes in the cluster.

Need to understand a bit better where you started from to be able to help.

Re: indexer replication

dkr3500 — Mon, 07 Jun 2021 21:31:35 GMT

@s2_splunk thanks for following-up on this post.

Splunk doc: https://docs.splunk.com/Documentation/Splunk/8.1.3/Indexer/Migratenon-clusteredindexerstoaclusteredenvironment

I needed to add:

repFactor = auto

under each [index_name] stanza on my cluster master's indexes.conf that gets pushed out to the peer nodes (indexers).

Once I updated indexes.conf, and applied the latest bundle on the CM (/opt/splunk/bin/splunk apply cluster-bundle --answer-yes), it started to replicated the custom indexes.

Important note: don't updated all indexes and push out the new cluster bundle at once, it its too much for the CM...update one index at a time and apply the cluster bundle.