Upgrade disk in Cluster

[jmsiegma]

I have 4 indexers in a cluster, that I need to upgrade the disk to gain more storage.

• All systems are Linux
• Each server in the cluster has an /app mount for splunk (/app/splunk/...)
• Each /app mount is 6TB of space with 4TB usable as RAID 6

I need to pull the existing drives, and replace them with larger drives to get more storage. The question is around what the best process is for the migration strategy:

Option 1) Back up the existing 4TB of data, Pull the existing disks, Replace with new larger disks, Rebuild the array on the new disks (16TB), Restore the files back to the new array, Let cluster re-sync, Repeat process for the other 3 servers.

Option 2) Pull the 4TB Raid Array, Install the new drives and configure RAID 16TB, and let the cluster rebuild what is missing, Once all the systems are Synced, Repeat process for other 3 servers.

The real question is, If you were to loose a whole server in an Index Cluster (On accident or on Purpose), would you loose 1/4 of the of the data, or would it get re-created once you rebuild the servers let the servers replicate things. This is making the assumption that we do NOT have a backup.

I ask the question, because I know in a cluster, some of the buckets get marked as Primary, and Secondary bucket.

Thanks

[mahamed_splunk]

The real question is, If you were to loose a whole server in an Index Cluster (On accident or on Purpose), would you loose 1/4 of the of the data,

This depends on your replication settings. If you have 2 copies of the data (replication_factor=2), then you will still have one copy in the system even after taking down one of the peers.

[mahamed_splunk]

Yes, it will replicate the data to 4th indexers whenever it joins the cluster.

[jmsiegma]

I have a replication_factor=4 in my 4 Indexer set. So if I removed the existing drives from the system, and replaced it with completely new drives and RAID set, I could then re-install splunk, and join the indexer to the cluster, and it would replicate 100% of the data to this new 4th instance. correct?