topic Re: What's the best way to perform maintenance on a member of a search head cluster? in Deployment Architecture

What's the best way to perform maintenance on a member of a search head cluster?

twinspop — Tue, 11 Oct 2016 22:26:11 GMT

I need to perform some emergency maintenance on 1 member of my 4-member Search Head Cluster tonight. From the docs, it looks like I need to remove the target from the SHC, clean the Splunk install, perform my maintenance (including a reboot), then re-add the target member back to the cluster. This seems insane to me. Is that really the best practice?

Would it be easier to just take down the entire cluster while working on this one machine?

Re: What's the best way to perform maintenance on a member of a search head cluster?

Steve_G_ — Tue, 11 Oct 2016 22:33:40 GMT

It seems to me that you should be able to just stop the instance, perform your maintenance, and then add it back into the cluster using this fairly simple procedure: http://docs.splunk.com/Documentation/Splunk/6.5.0/DistSearch/Addaclustermember#Add_a_member_that_left_the_cluster_without_being_removed_from_it

It shouldn't be necessary to perform the resync mentioned in that procedure if the maintenance occurs fairly quickly. In that case, you just need to restart the instance to add it back into the cluster.

Of course, it depends on what you mean by "maintenance". Presumably you won't be touching the Splunk configurations on the instance during the maintenance.

Re: What's the best way to perform maintenance on a member of a search head cluster?

jkat54 — Tue, 11 Oct 2016 22:35:49 GMT

Since its an existing member and you're only going to be "down" briefly and I assume you're not changing things under /etc/apps... I'd just do the maintenance and restart the bad boy.

Cons to this approach:
Could interfere with active users
Could interfere with summary indexing (hopefully you do summary indexing on a separate SHC or standalone SH)
Could force a re-election of the captai if it's the captain

Re: What's the best way to perform maintenance on a member of a search head cluster?

twinspop — Tue, 11 Oct 2016 22:46:21 GMT

So many 'shoulds' and 'coulds' in both answers. The uncertainty and finger crossing involved with SHC makes me very nervous. Some day this will improve, right? RIGHT?! 🙂

Re: What's the best way to perform maintenance on a member of a search head cluster?

jkat54 — Tue, 11 Oct 2016 23:46:42 GMT

Point was cowboy up! and reboot that thang! ...

lol to avoid the coulds, remove it first and therefore Steve's answer is most appropriate.

Re: What's the best way to perform maintenance on a member of a search head cluster?

Lucas_K — Wed, 12 Oct 2016 00:03:17 GMT

Also check the cluster master and move it if it happened to be the captain.

check who is the captain via "splunk show shcluster-status"

Relocate it to another host via "splunk transfer shcluster-captain -mgmt_uri https://myhostthatiwanttobethenewcaptain:8089". This should be executed on the current captain instance.

Re: What's the best way to perform maintenance on a member of a search head cluster?

twinspop — Wed, 12 Oct 2016 01:32:49 GMT

Good tip!

Re: What's the best way to perform maintenance on a member of a search head cluster?

twinspop — Wed, 12 Oct 2016 13:56:32 GMT

I chose the simply-stop-it method. The server was down for about 50 minutes. Restarted and just for grins ran splunk resync shcluster-replicated-config. It seems to be working okay.