Deployment Architecture

Why do we get not Ready for searchable rolling restart

Marlan
Explorer

We have a 8.1.2 dployment with 2 clustered indexers and a search head cluster. We have two master nodes, one is operational, the other is on stand-by.

When I run splunk show cluster-status --verbose on the master node I get all positive answers except for this one: "Ready for searchable rolling restart NO".

It is not explained in the documentation what is behind this check.

Does anyone know what could be the matter?

splunk show cluster-status --verbose

Pre-flight check successful .................. YES
Replication factor met ............... YES
Search factor met .................... YES
All data is searchable ............... YES
All peers are up ..................... YES
CM version is compatible ............. YES
No fixup tasks in progress ........... YES
Splunk version peer count { 8.1.2: 2 }
Ready for searchable rolling restart NO

Indexing Ready YES

idx14 6XYZ1234-9877-4911-A73E-XYZ12345FC43 default
Searchable YES
Status Up
Bucket Count=314
Splunk Version=8.1.2

idx13 DXYZ1234-8765-4DA5-BD4F-XYZ12345B8DF default
Searchable YES
Status Up
Bucket Count=316
Splunk Version=8.1.2

_audit
Number of non-site aware buckets=0
Number of buckets=124
Size=41491951
Searchable YES
Replicated copies tracker
122/124 122/124
Searchable copies tracker
122/124 122/124

_internal
Number of non-site aware buckets=0
Number of buckets=115
Size=3672105501
Searchable YES
Replicated copies tracker
115/115 115/115
Searchable copies tracker
115/115 115/115

_telemetry
Number of non-site aware buckets=0
Number of buckets=18
Size=67083
Searchable YES
Replicated copies tracker
18/18 18/18
Searchable copies tracker
18/18 18/18

main
Number of non-site aware buckets=0
Number of buckets=57
Size=6295363
Searchable YES
Replicated copies tracker
57/57 57/57
Searchable copies tracker
57/57 57/57

Labels (1)
0 Karma

codebuilder
Influencer

You have an even number of indexers in your cluster, and only two. If one of your indexers goes down (fails or restarts) then your data will not be searchable.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

Marlan
Explorer

Thanks for you reply.

Our indexes are fully replicated with a search factor of 2, so even if one indexer is down, data should be searchable after primary buckets have been re-assigned to the running indexer. A searchable rolling restart should do precisely that in advance of restarting the indexers, one at a time.

I cannot see why it should not work in our system.

https://docs.splunk.com/Documentation/Splunk/8.1.2/Indexer/Userollingrestart 

Unfortunately, the documentation offers no insight into the check that results in:
"Ready for searchable rolling restart NO"

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!