Deployment Architecture

How to tear down a search head cluster?

mgiddens
Path Finder

Good evening all.
I would like to know exactly how to properly tear down a search head cluster.

I am rebuilding / upgrading a Splunk environment I inherited and I need to re-utilize some of the servers in the seach head cluster for dedicated purposes; such as installed Enterprise security app on a dedicated search head.
I have thought this through long a hard and decided this is the direction I want to go.

I have already disabled search head cluster on the members (set disabled stanza to 1 in the shclustering stanzas.)
As soon as I did this, I am now getting KV store failed errors on all 3 search heads.
I did the KVstore status command and the status show failed on the status details.

I am not sure exactly where to go from here but I am sure that disabling search head clustering on my search heads caused the KV store issue.

How do you "undo/bring down" a search head cluster completely; or maybe its better to say "revert" the search head cluster back into individual search heads?
Could the KVStore errors really be associated with my disabling the search head members?
If not, where could this KV error be coming from?

Thank you.

mgiddens

skalliger
Motivator

Hi,

a rather short answer. If you're new to Splunk and inerhit an existing environment, Splunk has set-up a docs page describing how to get familiar with it.

Secondly, this page describes how to remove members from the cluster. After you're done, simple use one of those cleared instances as a standalone ES SH.

And yes, the KV store issues are most likely coming from those actions. I'd say bring the cluster up again to be working and then follow the procedure mentioned in the docs.

Skalli

0 Karma

mgiddens
Path Finder

Thanks for the response.
Turn the "disabled" setting back to 0 for all search heads; logged into one to verify and saw the "search head clustering" option available again in the UI.
Then tried to remove as you stated; no joy, still getting the KV errors.

It seems like once you get it into search head clustering, its hard to completely break it down so its as though you never configured it at all; which is what I am aiming to do.

Do I have to dis-engage the conf_deploy_fetch_uri (cluster deployer) too or any other conf file modifications on that server?

Any other ideas?

Thank you.

mgiddens

-Mike

0 Karma
Get Updates on the Splunk Community!

Technical Workshop Series: Splunk Data Management and SPL2 | Register here!

Hey, Splunk Community! Ready to take your data management skills to the next level? Join us for a 3-part ...

Spotting Financial Fraud in the Haystack: A Guide to Behavioral Analytics with Splunk

In today's digital financial ecosystem, security teams face an unprecedented challenge. The sheer volume of ...

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability As businesses scale ...