Deployment Architecture

Why do we get "not Ready for searchable rolling restart"?

Marlan
Explorer

We have a 8.1.2 dployment with 2 clustered indexers and a search head cluster. We have two master nodes, one is operational, the other is on stand-by.

When I run splunk show cluster-status --verbose on the master node I get all positive answers except for this one: "Ready for searchable rolling restart NO".

It is not explained in the documentation what is behind this check.

Does anyone know what could be the matter?

splunk show cluster-status --verbose

Pre-flight check successful .................. YES
Replication factor met ............... YES
Search factor met .................... YES
All data is searchable ............... YES
All peers are up ..................... YES
CM version is compatible ............. YES
No fixup tasks in progress ........... YES
Splunk version peer count { 8.1.2: 2 }
Ready for searchable rolling restart NO

Indexing Ready YES

idx14 6XYZ1234-9877-4911-A73E-XYZ12345FC43 default
Searchable YES
Status Up
Bucket Count=314
Splunk Version=8.1.2

idx13 DXYZ1234-8765-4DA5-BD4F-XYZ12345B8DF default
Searchable YES
Status Up
Bucket Count=316
Splunk Version=8.1.2

_audit
Number of non-site aware buckets=0
Number of buckets=124
Size=41491951
Searchable YES
Replicated copies tracker
122/124 122/124
Searchable copies tracker
122/124 122/124

_internal
Number of non-site aware buckets=0
Number of buckets=115
Size=3672105501
Searchable YES
Replicated copies tracker
115/115 115/115
Searchable copies tracker
115/115 115/115

_telemetry
Number of non-site aware buckets=0
Number of buckets=18
Size=67083
Searchable YES
Replicated copies tracker
18/18 18/18
Searchable copies tracker
18/18 18/18

main
Number of non-site aware buckets=0
Number of buckets=57
Size=6295363
Searchable YES
Replicated copies tracker
57/57 57/57
Searchable copies tracker
57/57 57/57

Labels (1)
0 Karma

codebuilder
SplunkTrust
SplunkTrust

You have an even number of indexers in your cluster, and only two. If one of your indexers goes down (fails or restarts) then your data will not be searchable.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

Marlan
Explorer

Thanks for you reply.

Our indexes are fully replicated with a search factor of 2, so even if one indexer is down, data should be searchable after primary buckets have been re-assigned to the running indexer. A searchable rolling restart should do precisely that in advance of restarting the indexers, one at a time.

I cannot see why it should not work in our system.

https://docs.splunk.com/Documentation/Splunk/8.1.2/Indexer/Userollingrestart 

Unfortunately, the documentation offers no insight into the check that results in:
"Ready for searchable rolling restart NO"

0 Karma

hrngr
Observer

Hello @Marlan ,

did you try to perform a rolling restart and if yes, did it work without downtime?

We have the same setup and are considerung to conduct a rolling upgrade of the cluster with only two indexers.

 

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

Usually it's better to create a new questions than use an old one.

What I have seen is that when the one has done rolling-restart for peers there is short time when previous node is not yet fully functional and the next one has started the restart. For that time when you have only two indexer, even with SF=2, all buckets (actually any in this case) are not searchable. I'm not sure how many nodes you must have to get working searchable rolling restart. My guess is that you should have at least 3 peer with SF=3, but haven't try this.

r. Ismo 

0 Karma

hrngr
Observer

Thanks. So in the case of a rolling upgrade that would be fine, because then it's in our hands to take down the second peer only after the first one is fully functional again.

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Exactly that way.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...