Deployment Architecture

Why do we get not Ready for searchable rolling restart

Marlan
Explorer

We have a 8.1.2 dployment with 2 clustered indexers and a search head cluster. We have two master nodes, one is operational, the other is on stand-by.

When I run splunk show cluster-status --verbose on the master node I get all positive answers except for this one: "Ready for searchable rolling restart NO".

It is not explained in the documentation what is behind this check.

Does anyone know what could be the matter?

splunk show cluster-status --verbose

Pre-flight check successful .................. YES
Replication factor met ............... YES
Search factor met .................... YES
All data is searchable ............... YES
All peers are up ..................... YES
CM version is compatible ............. YES
No fixup tasks in progress ........... YES
Splunk version peer count { 8.1.2: 2 }
Ready for searchable rolling restart NO

Indexing Ready YES

idx14 6XYZ1234-9877-4911-A73E-XYZ12345FC43 default
Searchable YES
Status Up
Bucket Count=314
Splunk Version=8.1.2

idx13 DXYZ1234-8765-4DA5-BD4F-XYZ12345B8DF default
Searchable YES
Status Up
Bucket Count=316
Splunk Version=8.1.2

_audit
Number of non-site aware buckets=0
Number of buckets=124
Size=41491951
Searchable YES
Replicated copies tracker
122/124 122/124
Searchable copies tracker
122/124 122/124

_internal
Number of non-site aware buckets=0
Number of buckets=115
Size=3672105501
Searchable YES
Replicated copies tracker
115/115 115/115
Searchable copies tracker
115/115 115/115

_telemetry
Number of non-site aware buckets=0
Number of buckets=18
Size=67083
Searchable YES
Replicated copies tracker
18/18 18/18
Searchable copies tracker
18/18 18/18

main
Number of non-site aware buckets=0
Number of buckets=57
Size=6295363
Searchable YES
Replicated copies tracker
57/57 57/57
Searchable copies tracker
57/57 57/57

Labels (1)
0 Karma

codebuilder
SplunkTrust
SplunkTrust

You have an even number of indexers in your cluster, and only two. If one of your indexers goes down (fails or restarts) then your data will not be searchable.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

Marlan
Explorer

Thanks for you reply.

Our indexes are fully replicated with a search factor of 2, so even if one indexer is down, data should be searchable after primary buckets have been re-assigned to the running indexer. A searchable rolling restart should do precisely that in advance of restarting the indexers, one at a time.

I cannot see why it should not work in our system.

https://docs.splunk.com/Documentation/Splunk/8.1.2/Indexer/Userollingrestart 

Unfortunately, the documentation offers no insight into the check that results in:
"Ready for searchable rolling restart NO"

0 Karma
Get Updates on the Splunk Community!

Welcome to the Future of Data Search & Exploration

You have more data coming at you than ever before. Over the next five years, the total amount of digital data ...

What’s new on Splunk Lantern in August

This month’s Splunk Lantern update gives you the low-down on all of the articles we’ve published over the past ...

This Week's Community Digest - Splunk Community Happenings [8.3.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...