I am going to have to tear down our SHC; I had to give up our 3rd search head to another project, which puts us in un-supported land (2 node cluster). I plan to split the deployed apps between the 2 remaining search heads.
Right now, I plan is to back up $SPLUNK_HOME/etc/apps on each of the two remaining nodes, remove the [shcluster] stanzas, then go through all the apps, comparing the contents of /etc/apps//local and /etc/apps//default on the search heads with what was getting pushed from the deployer, making sure that the default on the search heads matches the default from the deployers, and the changes in local on the deployers are merged into the local changes on the search heads.
Have you valorate the option of make 2 more instances of search head in the two servers, to have 4 search head in 2 machines?
Just to keep the search head cluster?
Hope help you.
If you have a local configuration in the search head this will no apply
"making sure that the default on the search heads matches the default from the deployers" couse the changes in the shc are not pushing to the deployer.
why won't this apply? won't I want both the local changes from the deployer, and any local changes that have been made during the operation of the cluster?
As you know, all the files in the deployer from default and local are merge to the default folder in ther search head. So you only need to check the default and local in ther search head.
Hope help you.
I am trying to get the applications back to a state of original install. If I tear the cluster down and leave local changes (from the deployer) in default.conf, I will lose them in the next upgrade....