Our splunk indexers disk has reached 90% and decided to extend the capacity. We have 3 indexers in cluster . What are the steps to be followed to increase the capacity. Indexers are linux based and the replication factor is 3.
1. Can we increase the storage on the fly without any downtime
2. Will it affect the existing buckets? Any backup is required.
If you are using LVM , it is possible increasing partition on the fly. Please check with your system admin. This will not effect the buckets. Otherwise you may need reboot the system or create a new partition and copy the buckets to new partition.
In any case, it is better enable maintenance mode first, doing resize operation on one indexer at a time and disabling maintenance mode after each indexer to allow cluster master make necessary fix ups. This will also prevent data loss in case of a failure on one indexer.