Deployment Architecture

Issues with rolling restart (index cluster) & idea to add additional index peers

davidjohnbecket
Path Finder

We have in index cluster:

  • two node index cluster
  • two sites (one index peer in each site)

We are seeing an issue whereby rolling restarts of our index cluster is causing indexing issues on the forwarders as the to peers are restarting. One index peer restart is triggered and before its fully operational the second index peer restarts.
I think this is because we only have a single index peer in each site and the rolling restart is managed at the site level.

We currently have a the following configuration on the Master Node:

mode = master
multisite=true
available_sites=site1,site2
site_replication_factor = origin:1,site1:1,site2:1,total:2
site_search_factor = origin:1,site1:1,site2:1,total:2
replication_factor = 1
search_factor = 1

My intention is to add an additional index peer to each site but maintaining the current replication factor and search factor.

The reason for this would be to maintain the data replication factor without increasing the storage requirements substantially but allow us to performing rolling restarts more effectively

Are my assumptions and this approach correct?
Would i be able to rebalance the data between the old and new index peers?
What have i missed form this idea?

This is what my environment looks like currently and with the additional index peers.

alt text

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...