Splunk shcluster configuration on master not updating config

[brettcave]

I am trying to configure a master node in a splunk cluster to be an indexer master and shc deployer. I install Splunk Enterprise (7.3.1) and then execute commands to configure it:

splunk edit shcluster-config -secret clustersecret -shcluster_label myshclabel -auth user:pass

After doing this, etc/system/local/server.conf is updated with a [shcluster] heading, but no pass4SymmKey or cluster label is configured. it is a blank stanza.

I am also running edit cluster-config on the server and this is updating the config as expected. My understanding is that these commands should add a cluster label and secret to the config. Am I missing something? Not sure if licensing needs to be configured first? For my use-case, manually editing the file is not an option, it needs to be configured by command.

[jdhunter · Accepted solution]

Do you have a License installed on this server? I would setup a License Master and point all of your Splunk instances to it.

Once you have a license master configured, you can run this command on all your Splunk servers.

Setting License Master Slave

splunk edit licenser-localslave -master_uri 'https://license_master_uri:8089'

https://docs.splunk.com/Documentation/Splunk/7.3.1/Admin/Configurealicensemaster
https://docs.splunk.com/Documentation/Splunk/7.3.1/Admin/LicenserCLIcommands

Next, I would setup the Cluster Master and the Cluster Peers

Cluster Master

splunk edit cluster-config -mode master -replication_factor # -search_factor # -secret your_key -cluster_label cluster1

Cluster Peer (Your Indexers)

splunk edit cluster-config -mode slave -master_uri https://cluster_master:8089 -replication_port 9887 -secret your_key

https://docs.splunk.com/Documentation/Splunk/7.3.1/Indexer/Enablethemasternode
https://docs.splunk.com/Documentation/Splunk/7.3.1/Indexer/ConfiguremasterwithCLI
https://docs.splunk.com/Documentation/Splunk/7.3.1/Indexer/ConfigurepeerswithCLI

Finally, there are multiple steps in setting up a Search Head Cluster to Deploy the cluster:

These are the key steps in deploying clusters:

1. Identify your requirements.

2. Set up the deployer.

3. Install the Splunk Enterprise instances.

4. Initialize cluster members.

5. Bring up the cluster captain.

6. Perform post-deployment set-up.

https://docs.splunk.com/Documentation/Splunk/7.3.1/DistSearch/SHCdeploymentoverview

System Requirements for installing Cluster Master and SHC Deployer on the same machine

https://docs.splunk.com/Documentation/Splunk/7.3.1/Indexer/Systemrequirements

[brettcave]

@jdhunter - thanks for bringing up the license question - I was not installing the license. I have updated it to do so, but am still not seeing the right config get added.

$ splunk add licenses /path/to/enterprise.lic 
The licenses object has been added
You need to restart the Splunk Server (splunkd) for your changes to take effect.
$ service splunk restart
$ splunk list licenses # enterprise license shows.
$ splunk list licenser-pools
    auto_generated_pool_enterprise
        description:auto_generated_pool_enterprise
        effective_quota:REDACTED
        is_unlimited:0
        quota:MAX
        slaves:
        stack_id:enterprise
        used_bytes:0
$ splunk edit shcluster-config -secret testingsecret  -shcluster_label shclusterlabel_test
The shcluster-config property has been edited.
$ cat etc/system/local/server.conf
[lmpool:auto_generated_pool_enterprise]
description = auto_generated_pool_enterprise
quota = MAX
slaves = *
stack_id = enterprise

[license]
active_group = Enterprise

[shclustering]

shcluster_label and the secret is not set still in the [shclustering] section. However, when I run the indexer cluster master commands "edit cluster-config" with cluster_label and secret, it is set as expected in the [clustering] section.

[brettcave]

A quick note: splunk help edit shcluster-config does not show shcluster_label in the options - it shows secret and then replication factor param, timeouts and port settings.

Also - I can set labels and secrets fine for the indexer cluster master and cluster. I am trying to also configure the indexer master as the shc deployer, it's the shcluster-config command that is not updating the config. However, I am still able to bring up shc's in a shc cluster with no label specified and it connects / registers to the deployer fine. However, i'd like to specify a secret and a label

[brettcave]

I am using this to configure shc deployer: https://docs.splunk.com/Documentation/Splunk/7.3.1/DistSearch/SHCconfigurationoverview#Configuration...

[brettcave]

last post: this doc describes setting up the deployer in step 2 - https://docs.splunk.com/Documentation/Splunk/7.3.1/DistSearch/SHCdeploymentoverview

however, i am looking to use the CLI to configure it, not modifying config files directly.

[jdhunter]

From my clustering class notes, there is no CLI support for enabling the deployer. You have to edit the server.conf file and then restart splunk:

[shclustering]
pass4SymmKey =

Secret needs to be the same on deployer and SHC members

[brettcave]

thanks. accepting your answer.