Deployment Architecture

Issues with rolling restart (index cluster) & idea to add additional index peers

davidjohnbecket
Path Finder

We have in index cluster:

  • two node index cluster
  • two sites (one index peer in each site)

We are seeing an issue whereby rolling restarts of our index cluster is causing indexing issues on the forwarders as the to peers are restarting. One index peer restart is triggered and before its fully operational the second index peer restarts.
I think this is because we only have a single index peer in each site and the rolling restart is managed at the site level.

We currently have a the following configuration on the Master Node:

mode = master
multisite=true
available_sites=site1,site2
site_replication_factor = origin:1,site1:1,site2:1,total:2
site_search_factor = origin:1,site1:1,site2:1,total:2
replication_factor = 1
search_factor = 1

My intention is to add an additional index peer to each site but maintaining the current replication factor and search factor.

The reason for this would be to maintain the data replication factor without increasing the storage requirements substantially but allow us to performing rolling restarts more effectively

Are my assumptions and this approach correct?
Would i be able to rebalance the data between the old and new index peers?
What have i missed form this idea?

This is what my environment looks like currently and with the additional index peers.

alt text

0 Karma
Get Updates on the Splunk Community!

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...

Want to Reduce Costs, Mitigate Risk, Improve Performance, or Increase Efficiencies? ...

Splunk Lantern is Splunk’s customer success center that provides advice from Splunk experts on valuable data ...