Why can't I re-enable or re-add one member of my search head cluster after upgrading it from 6.4 to 6.5.0?

[lycollicott]

I am following the upgrade instructions at http://docs.splunk.com/Documentation/Splunk/6.5.0/Installation/UpgradeyourdistributedSplunkEnterpris... like so:

Upgrade the search heads
1. Disable one of the search heads.
2. Upgrade the search head. Do not let it restart.
3. After you upgrade the search head, place the confirmed working apps into the $SPLUNK_HOME/etc/apps directory of the search head.
4. Re-enable and restart the search head.
5. Test apps on the search head for operation and functionality.
6. If there are no problems with the search head, then disable and upgrade the remaining search heads, one by one. Repeat this step until you have reached the last search head in your environment.
7. (Optional) Test each search head for operation and functionality after you bring it up.
8. After you upgrade the last search head, test all of the search heads for operation and functionality.

At step 1 I did splunk disable shcluster-config but I have a feeling I screwed that up, because at step 4 I cannot re-enable it with either splunk add shcluster-member -new_member_uri https://blah:8089 or with splunk add shcluster-member -current_member_uri https://blah:8089

What have I messed up on this sunny Friday afternoon?

[lycollicott · Accepted solution]

Ok, here is what I did this evening to fix it (I was over complicating it earlier, but it became clearer after I took a break for a few hours.)

splunk init shcluster-config -auth admin:<password> -mgmt_uri https://sh1.domain.com:8089 -replication_port 8999 -replication_factor 3 -conf_deploy_fetch_url https://ds.domain.com:8089 -secret <something>-shcluster_label prodsearch1

splunk restart

splunk add shcluster-member -current_member_uri https://sh2.domain.com:8089

[ChrisG]

I'm no expert in this procedure, but I do know that the instructions for upgrading a search head cluster are here:

http://docs.splunk.com/Documentation/Splunk/6.5.0/DistSearch/UpgradeaSHC

[lycollicott]

Oh, if only the other 6.5.0 manual had said that I could have avoided the disable step. Bummer.

[ChrisG]

I found it by following a link in the topic you posted....