I have read out the splunk document for Upgrade a 6.x indexer cluster to a later version of 6.x , In the document they mention the below steps to perform the up gradation.
Stop the master.
Stop all the peers and search heads.
When bringing down the peers, use the splunk stop command, not splunk offline.
Upgrade the master node, following the normal procedure for any Splunk Enterprise upgrade, as described in How to upgrade Splunk Enterprise in the Installation Manual. Do not upgrade the peers yet.
Start the master, accepting all prompts, if it is not already running.
Run splunk enable maintenance-mode on the master. To confirm that the master is in maintenance mode, run splunk show maintenance-mode. This step prevents unnecessary bucket fix-ups. See Use maintenance mode.
Upgrade the peer nodes and search heads, following the normal procedure for any Splunk Enterprise upgrade, as described in How to upgrade Splunk Enterprise in the Installation Manual.
If the search heads in the indexer cluster are members of a search head cluster, see Upgrade a search head cluster.
Start the peer nodes and search heads, if they are not already running.
Run splunk disable maintenance-mode on the master. To confirm that the master is not in maintenance mode, run splunk show maintenance-mode.
If we stop the all the indexer at a one time, will get the data loss.
Your forwarders should buffer input while the indexers are down.
With default configuration on your forwarders, that will be very limited right?
You'd need to enable appropriately sized persistent queues to allow for the amount of downtime that you would need for such an upgrade procedure, right? And such persistent queues only work for certain input types as well.
But I guess unless you have a multisite indexer cluster, there is no way to do an upgrade without indeed having your whole cluster down temporarily. Which depending on the configuration of your data sources, potential aggregators like syslog servers and your forwarders, that could cause data loss.
@lksridhar: what exactly is your question though?