After upgrading from 9.0.3 version to 9.1.0 Version facing the below Error (in Cluster Master, Search head)

[Sathish28]

Invalid key in stanza [clustermaster:one] in /apps/splunk/splunk/etc/apps/100_gnw_cluster_search_base/local/server.conf,
line 7: master_uri (value: https://<address>:8089).

Invalid key in stanza [clustermaster:one] in /apps/splunk/splunk/etc/apps/100_gnw_cluster_search_base/local/server.conf,
line 8: pass4SymmKey (value: ***************************************).

Invalid key in stanza [clustermaster:one] in /apps/splunk/splunk/etc/apps/100_gnw_cluster_search_base/local/server.conf,
line 9: multisite (value: true)

[Sathish28]

Thank you @PickleRick , Changed the master to Manager in Cluster and URi
which worked

[PickleRick · Accepted solution]

My suspicion would be that you're using mixed terminology the master/slave terms have been obsoleted several big versions ago. Now it's manager/peer so you should use clustermanager stanza and manager_uri setting.

[SanjayReddy]

Hi @Sathish28 

Can you please run following comand in CLI of the server , where you are seeing in message and share the output.

/opt/splunk/bin/splunk btool server list  --debug | grep -i local 

[Sathish28]

In Cluster Master : I find the below search

 splunk btool server list --debug | grep -i local/apps/splunk/splunk/etc/apps/100_gnw_cluster_master_base/local/server.conf [clustering]
/apps/splunk/splunk/etc/apps/100_gnw_cluster_master_base/local/server.conf available_sites = site1
/apps/splunk/splunk/etc/system/local/server.conf maintenance_mode = false
/apps/splunk/splunk/etc/apps/100_gnw_cluster_search_base/local/server.conf master_uri = clustermaster:one
/apps/splunk/splunk/etc/apps/100_gnw_cluster_master_base/local/server.conf mode = master
/apps/splunk/splunk/etc/apps/100_gnw_cluster_master_base/local/server.conf multisite = true
/apps/splunk/splunk/etc/apps/100_gnw_cluster_master_base/local/server.conf pass4SymmKey = **************
/apps/splunk/splunk/etc/apps/100_gnw_cluster_master_base/local/server.conf replication_factor = 2
/apps/splunk/splunk/etc/apps/100_gnw_cluster_master_base/local/server.conf search_factor = 1
/apps/splunk/splunk/etc/apps/100_gnw_cluster_master_base/local/server.conf site_replication_factor = origin:1, total:2
/apps/splunk/splunk/etc/apps/100_gnw_cluster_master_base/local/server.conf site_search_factor = origin:1, total:2
/apps/splunk/splunk/etc/apps/100_gnw_cluster_search_base/local/server.conf [clustermaster:one]
/apps/splunk/splunk/etc/apps/100_gnw_cluster_search_base/local/server.conf master_uri = https:webaddress:8089
/apps/splunk/splunk/etc/apps/100_gnw_cluster_search_base/local/server.conf multisite = true
/apps/splunk/splunk/etc/apps/100_gnw_cluster_search_base/local/server.conf pass4SymmKey = *****************
/apps/splunk/splunk/etc/apps/100_gnw_cluster_master_base/local/server.conf [general]
/apps/splunk/splunk/etc/system/local/server.conf pass4SymmKey = ****************
/apps/splunk/splunk/etc/system/local/server.conf serverName = webaddress
/apps/splunk/splunk/etc/apps/100_gnw_cluster_master_base/local/server.conf site = site1
/apps/splunk/splunk/etc/system/local/server.conf [kvstore]
/apps/splunk/splunk/etc/apps/100_gnw_license_master/local/server.conf [license]
/apps/splunk/splunk/etc/apps/100_gnw_license_master/local/server.conf master_uri = https:webaddress:8089
/apps/splunk/splunk/etc/system/local/server.conf [lmpool:auto_generated_pool_download-trial]
/apps/splunk/splunk/etc/system/local/server.conf description = auto_generated_pool_download-trial
/apps/splunk/splunk/etc/system/local/server.conf quota = MAX
/apps/splunk/splunk/etc/system/local/server.conf slaves = *
/apps/splunk/splunk/etc/system/local/server.conf stack_id = download-trial
/apps/splunk/splunk/etc/system/local/server.conf [lmpool:auto_generated_pool_forwarder]
/apps/splunk/splunk/etc/system/local/server.conf description = auto_generated_pool_forwarder
/apps/splunk/splunk/etc/system/local/server.conf quota = MAX
/apps/splunk/splunk/etc/system/local/server.conf slaves = *
/apps/splunk/splunk/etc/system/local/server.conf stack_id = forwarder
/apps/splunk/splunk/etc/system/local/server.conf [lmpool:auto_generated_pool_free]
/apps/splunk/splunk/etc/system/local/server.conf description = auto_generated_pool_free
/apps/splunk/splunk/etc/system/local/server.conf quota = MAX
/apps/splunk/splunk/etc/system/local/server.conf slaves = *
/apps/splunk/splunk/etc/system/local/server.conf stack_id = free
/apps/splunk/splunk/etc/system/default/server.conf alert_store = local
/apps/splunk/splunk/etc/system/default/server.conf suppression_store = local
/apps/splunk/splunk/etc/system/default/server.conf conf_replication_summary.includelist.refine.local = (system|(apps/*)|users(/_reserved)?/*/*)/(local/...|metadata/local.meta)
/apps/splunk/splunk/etc/system/local/server.conf [sslConfig]
/apps/splunk/splunk/etc/system/local/server.conf sslPassword = ***********************

In Deployment Server : I find the below Search

/apps/splunk/splunk/etc/apps/100_gnw_cluster_search_base/local/server.conf [clustering]
/apps/splunk/splunk/etc/apps/100_gnw_cluster_search_base/local/server.conf master_uri = clustermaster:one
/apps/splunk/splunk/etc/apps/100_gnw_cluster_search_base/local/server.conf mode = searchhead
/apps/splunk/splunk/etc/apps/100_gnw_cluster_search_base/local/server.conf [clustermaster:one]
/apps/splunk/splunk/etc/apps/100_gnw_cluster_search_base/local/server.conf master_uri = https://webaddress:8089
/apps/splunk/splunk/etc/apps/100_gnw_cluster_search_base/local/server.conf multisite = true
/apps/splunk/splunk/etc/apps/100_gnw_cluster_search_base/local/server.conf pass4SymmKey = *************************
/apps/splunk/splunk/etc/apps/100_gnw_cluster_search_base/local/server.conf [general]
/apps/splunk/splunk/etc/system/local/server.conf pass4SymmKey = *************************
/apps/splunk/splunk/etc/system/local/server.conf serverName = webaddress
/apps/splunk/splunk/etc/apps/100_gnw_cluster_search_base/local/server.conf site = site1
/apps/splunk/splunk/etc/system/local/server.conf [kvstore]
/apps/splunk/splunk/etc/system/local/server.conf [license]
/apps/splunk/splunk/etc/system/local/server.conf master_uri = https://webaddress:8089
/apps/splunk/splunk/etc/system/local/server.conf [lmpool:auto_generated_pool_download-trial]
/apps/splunk/splunk/etc/system/local/server.conf description = auto_generated_pool_download-trial
/apps/splunk/splunk/etc/system/local/server.conf quota = MAX
/apps/splunk/splunk/etc/system/local/server.conf slaves = *
/apps/splunk/splunk/etc/system/local/server.conf stack_id = download-trial
/apps/splunk/splunk/etc/system/local/server.conf [lmpool:auto_generated_pool_forwarder]
/apps/splunk/splunk/etc/system/local/server.conf description = auto_generated_pool_forwarder
/apps/splunk/splunk/etc/system/local/server.conf quota = MAX
/apps/splunk/splunk/etc/system/local/server.conf slaves = *
/apps/splunk/splunk/etc/system/local/server.conf stack_id = forwarder
/apps/splunk/splunk/etc/system/local/server.conf [lmpool:auto_generated_pool_free]
/apps/splunk/splunk/etc/system/local/server.conf description = auto_generated_pool_free
/apps/splunk/splunk/etc/system/local/server.conf quota = MAX
/apps/splunk/splunk/etc/system/local/server.conf slaves = *
/apps/splunk/splunk/etc/system/local/server.conf stack_id = free
/apps/splunk/splunk/etc/system/default/server.conf alert_store = local
/apps/splunk/splunk/etc/system/default/server.conf suppression_store = local
/apps/splunk/splunk/etc/system/default/server.conf conf_replication_summary.includelist.refine.local = (system|(apps/*)|users(/_reserved)?/*/*)/(local/...|metadata/local.meta)
/apps/splunk/splunk/etc/system/local/server.conf [sslConfig]
/apps/splunk/splunk/etc/system/local/server.conf sslPassword = *************************

[PickleRick]

1. CM does not use master_uri (or manager_uri - master_uri is a deprecated setting) unless you're using failover CM. You don't seem to use one here so you don't need this setting (as well as whole stanzas defining those managers)

2. I'm not sure if you're listing settings from deployer or deployment server (if you have SHC, you must use deployer; single SHs can indeed be pushed from DS but is it your situation?). In either case, apps destined for SH(s) should be either put into $SPLUNK_HOME/shcluster or $SPLUNK_HOME/deployment-apps

The whole setup seems a bit strange.