Why do we get "not Ready for searchable rolling restart"?

[Marlan]

We have a 8.1.2 dployment with 2 clustered indexers and a search head cluster. We have two master nodes, one is operational, the other is on stand-by.

When I run splunk show cluster-status --verbose on the master node I get all positive answers except for this one: "Ready for searchable rolling restart NO".

It is not explained in the documentation what is behind this check.

Does anyone know what could be the matter?

splunk show cluster-status --verbose

Pre-flight check successful .................. YES
Replication factor met ............... YES
Search factor met .................... YES
All data is searchable ............... YES
All peers are up ..................... YES
CM version is compatible ............. YES
No fixup tasks in progress ........... YES
Splunk version peer count { 8.1.2: 2 }
Ready for searchable rolling restart NO

Indexing Ready YES

idx14 6XYZ1234-9877-4911-A73E-XYZ12345FC43 default
Searchable YES
Status Up
Bucket Count=314
Splunk Version=8.1.2

idx13 DXYZ1234-8765-4DA5-BD4F-XYZ12345B8DF default
Searchable YES
Status Up
Bucket Count=316
Splunk Version=8.1.2

_audit
Number of non-site aware buckets=0
Number of buckets=124
Size=41491951
Searchable YES
Replicated copies tracker
122/124 122/124
Searchable copies tracker
122/124 122/124

_internal
Number of non-site aware buckets=0
Number of buckets=115
Size=3672105501
Searchable YES
Replicated copies tracker
115/115 115/115
Searchable copies tracker
115/115 115/115

_telemetry
Number of non-site aware buckets=0
Number of buckets=18
Size=67083
Searchable YES
Replicated copies tracker
18/18 18/18
Searchable copies tracker
18/18 18/18

main
Number of non-site aware buckets=0
Number of buckets=57
Size=6295363
Searchable YES
Replicated copies tracker
57/57 57/57
Searchable copies tracker
57/57 57/57

[codebuilder]

You have an even number of indexers in your cluster, and only two. If one of your indexers goes down (fails or restarts) then your data will not be searchable.

[Marlan]

Thanks for you reply.

Our indexes are fully replicated with a search factor of 2, so even if one indexer is down, data should be searchable after primary buckets have been re-assigned to the running indexer. A searchable rolling restart should do precisely that in advance of restarting the indexers, one at a time.

I cannot see why it should not work in our system.

https://docs.splunk.com/Documentation/Splunk/8.1.2/Indexer/Userollingrestart 

Unfortunately, the documentation offers no insight into the check that results in:
"Ready for searchable rolling restart NO"

[hrngr]

Hello @Marlan ,

did you try to perform a rolling restart and if yes, did it work without downtime?

We have the same setup and are considerung to conduct a rolling upgrade of the cluster with only two indexers.

[isoutamo]

Hi

Usually it's better to create a new questions than use an old one.

What I have seen is that when the one has done rolling-restart for peers there is short time when previous node is not yet fully functional and the next one has started the restart. For that time when you have only two indexer, even with SF=2, all buckets (actually any in this case) are not searchable. I'm not sure how many nodes you must have to get working searchable rolling restart. My guess is that you should have at least 3 peer with SF=3, but haven't try this.

r. Ismo 

[hrngr]

Thanks. So in the case of a rolling upgrade that would be fine, because then it's in our hands to take down the second peer only after the first one is fully functional again.

[isoutamo]

Exactly that way.