Why are not all indexes searchable when i have 1 broken indexer out of 2?

[Gryphus]

I have 2 indexers in a cluster. One is down and one is up. All buckets are there on the indexer that is up but still not all indexes are searchable. Why is this and what can i do?

[Gryphus]

Thanks for your suggestion but It does not seem to be repFactor. Its set on Auto for all indexes.

[splunk@servername~]$ splunk btool indexes list _internal | grep repFactor
repFactor = auto

[kiran_panchavat]

@Gryphus 

Since only 1 is available, it's not fully searchable, meaning the search factor is not met. However, all data should still be searchable, as there is at least one searchable copy. With one indexer down, the search factor of 2 isn't met, as only one searchable copy is available. This makes indexes not fully searchable, but searches should still work with the up indexer's data.

In your case, you have a 2-node cluster, and based on the details, both the replication factor and search factor appear to be set to 2. This means:

• Each bucket (the basic unit of index storage) should have a primary copy on one indexer and a replica on the other.
• Both copies are designated as searchable to meet the search factor of 2.

https://community.splunk.com/t5/Deployment-Architecture/Is-it-possible-that-Search-Factor-is-Not-Met... 

[PickleRick]

You are wrong here. You're mixing rep/search factor with index searchability.

https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Howtomonitoracluster#Indexes_tab

"Fully searchable. Is the index fully searchable? In other words, does it have at least one searchable copy of each bucket? If even one bucket in the index does not have a searchable copy, this field will report the index as non-searchable."

About the original issue, the way to go would probably be to check buckets status - there should be a button on that tab to see buckets status in detail.

As a side note - indexer cluster with just two nodes isn't really fault-tolerant. It's kinda like RAID-1 without a hot-spare - when you lose one node, you're in degraded state and you don't have anywhere to replicate to.

[gcusello]

Hi @Gryphus ,

are only thise three indexes not fully searchable or all the indexes?

Ciao.

Giuseppe

[Gryphus]

It was about ~20 of 100 indexes that where not fully searchable. We have fixed the hardware issues on node2 so everything is repairing now.

[gcusello]

Hi @Gryphus ,

you should analyze the indexes.conf in your cluster Manager (and so in your Indexers) and verify if you configured 

repFactor = auto

for those indexes, as described at https://docs.splunk.com/Documentation/Splunk/latest/Admin/Indexesconf

Ciao.

Giuseppe