How to tear down a search head cluster?

[mgiddens]

Good evening all.
I would like to know exactly how to properly tear down a search head cluster.

I am rebuilding / upgrading a Splunk environment I inherited and I need to re-utilize some of the servers in the seach head cluster for dedicated purposes; such as installed Enterprise security app on a dedicated search head.
I have thought this through long a hard and decided this is the direction I want to go.

I have already disabled search head cluster on the members (set disabled stanza to 1 in the shclustering stanzas.)
As soon as I did this, I am now getting KV store failed errors on all 3 search heads.
I did the KVstore status command and the status show failed on the status details.

I am not sure exactly where to go from here but I am sure that disabling search head clustering on my search heads caused the KV store issue.

How do you "undo/bring down" a search head cluster completely; or maybe its better to say "revert" the search head cluster back into individual search heads?
Could the KVStore errors really be associated with my disabling the search head members?
If not, where could this KV error be coming from?

Thank you.

mgiddens

[skalliger]

Hi,

a rather short answer. If you're new to Splunk and inerhit an existing environment, Splunk has set-up a docs page describing how to get familiar with it.

Secondly, this page describes how to remove members from the cluster. After you're done, simple use one of those cleared instances as a standalone ES SH.

And yes, the KV store issues are most likely coming from those actions. I'd say bring the cluster up again to be working and then follow the procedure mentioned in the docs.

Skalli

[mgiddens]

Thanks for the response.
Turn the "disabled" setting back to 0 for all search heads; logged into one to verify and saw the "search head clustering" option available again in the UI.
Then tried to remove as you stated; no joy, still getting the KV errors.

It seems like once you get it into search head clustering, its hard to completely break it down so its as though you never configured it at all; which is what I am aiming to do.

Do I have to dis-engage the conf_deploy_fetch_uri (cluster deployer) too or any other conf file modifications on that server?

Any other ideas?

Thank you.

mgiddens

-Mike