I’m running a mixed cluster where my CM is 6.2.2 and my indexer peers are all running 6.1.4. One of the peers in the cluster crashed which resulted in my search and replication factors to both be off. I could see fixup was running on the indexes so I gave it sometime to complete. After a day i reviewed the status of the indexes in SOS and fixup showed that it was successfully run on all the indexes. Checked the Indexer Clustering view and both the search and replication factors showed as being met. However, non of the indexes were showing as "Fully Searchable". I gave it more time to complete. Still no change. I then performed a complete restart of the cluster. Still no change. The logs showed that all was good in the cluster. Why won’t the cluster show as fully searchable?
What happens with the already indexed logs after changing mode to mixed ?
As I stated before my cluster was running in "Mixed Mode". When running in mixed mode the following setting needs to be set in the server.conf file on the cluster master:
use_batch_mask_changes = false
After changing this setting and restarting the Cluster Master, all the indexes showed as fully searchable.
This is actually documented as a required change when running in mixed mode.
http://docs.splunk.com/Documentation/Splunk/6.2.2/Indexer/Systemrequirements