We have multi-site cluster, with four indexers on each site. I guess a replication factor of 2 is fine so each event resides on each site. We wonder about the search factor. Can we use one? In case of an indexer going down, what would happen with replication at 2 and search at 1?
The dilemma we have is that, when we experimented with 2:2, the disk usage was significantly higher than 2:1.
This depends on the amount of disk you have available and are willing to give up. When you do a Rep_factor of 2 and search_factor of 2 it is keeping both the raw data and the index files on the both indexers it is replicating to. When you have a search_factor of 1 it is keeping raw and index files on one indexer and then just replicating the index files to the other indexer. This saves on space and provides enough data to rebuild the index if needed.
On a multi-site cluster I would do at least origin:2,total:3. replication This will provide you site1: indexer with raw&index files, indexer with index files and site2: indexer with index files
If you do a search factor of origin1:total2 this will make the site2 indexer maintain both raw and index files.
Take a few moments to review this Splunk Doc as it goes in more detail on setting up search and replication factors on a mulitsite cluster.
Thank you @kgderrekchapin.
We really wonder about -
-- The dilemma we have is that, when we experimented with 2:2, the disk usage was significantly higher than 2:1.
So, let's say we are at 2:1 and one indexer goes down. What's the process to make the missing data searchable? how long would it take?
First let me correct myself. only the raw data is replicated. So in the above example you can swap the terms raw and index files.
Here is what happens when you lose a replicated index in at 2:1 scenario. When the searchable index goes down. the replicated index will then start to reindex from the raw data. The time it takes to make it fully searchable again is unknown as it depends on multitude of factors. But to reach that "fully searchable" threshold it will have to finish reindexing the all of the buckets in that index.