Deployment Architecture
Highlighted

Splunk shcluster configuration on master not updating config

Builder

I am trying to configure a master node in a splunk cluster to be an indexer master and shc deployer. I install Splunk Enterprise (7.3.1) and then execute commands to configure it:

splunk edit shcluster-config -secret clustersecret -shcluster_label myshclabel -auth user:pass

After doing this, etc/system/local/server.conf is updated with a [shcluster] heading, but no pass4SymmKey or cluster label is configured. it is a blank stanza.

I am also running edit cluster-config on the server and this is updating the config as expected. My understanding is that these commands should add a cluster label and secret to the config. Am I missing something? Not sure if licensing needs to be configured first? For my use-case, manually editing the file is not an option, it needs to be configured by command.

Tags (2)
0 Karma
Highlighted

Re: Splunk shcluster configuration on master not updating config

Path Finder

Do you have a License installed on this server? I would setup a License Master and point all of your Splunk instances to it.

Once you have a license master configured, you can run this command on all your Splunk servers.

Setting License Master Slave

splunk edit licenser-localslave -masteruri 'https://licensemaster_uri:8089'

https://docs.splunk.com/Documentation/Splunk/7.3.1/Admin/Configurealicensemaster
https://docs.splunk.com/Documentation/Splunk/7.3.1/Admin/LicenserCLIcommands

Next, I would setup the Cluster Master and the Cluster Peers

Cluster Master

splunk edit cluster-config -mode master -replicationfactor # -searchfactor # -secret yourkey -clusterlabel cluster1

Cluster Peer (Your Indexers)

splunk edit cluster-config -mode slave -masteruri https://clustermaster:8089 -replicationport 9887 -secret yourkey

https://docs.splunk.com/Documentation/Splunk/7.3.1/Indexer/Enablethemasternode
https://docs.splunk.com/Documentation/Splunk/7.3.1/Indexer/ConfiguremasterwithCLI
https://docs.splunk.com/Documentation/Splunk/7.3.1/Indexer/ConfigurepeerswithCLI

Finally, there are multiple steps in setting up a Search Head Cluster to Deploy the cluster:

These are the key steps in deploying clusters:

  1. Identify your requirements.

  2. Set up the deployer.

  3. Install the Splunk Enterprise instances.

  4. Initialize cluster members.

  5. Bring up the cluster captain.

  6. Perform post-deployment set-up.

https://docs.splunk.com/Documentation/Splunk/7.3.1/DistSearch/SHCdeploymentoverview

System Requirements for installing Cluster Master and SHC Deployer on the same machine

https://docs.splunk.com/Documentation/Splunk/7.3.1/Indexer/Systemrequirements

View solution in original post

Highlighted

Re: Splunk shcluster configuration on master not updating config

Builder

@jdhunter - thanks for bringing up the license question - I was not installing the license. I have updated it to do so, but am still not seeing the right config get added.

$ splunk add licenses /path/to/enterprise.lic 
The licenses object has been added
You need to restart the Splunk Server (splunkd) for your changes to take effect.
$ service splunk restart
$ splunk list licenses # enterprise license shows.
$ splunk list licenser-pools
    auto_generated_pool_enterprise
        description:auto_generated_pool_enterprise
        effective_quota:REDACTED
        is_unlimited:0
        quota:MAX
        slaves:
        stack_id:enterprise
        used_bytes:0
$ splunk edit shcluster-config -secret testingsecret  -shcluster_label shclusterlabel_test
The shcluster-config property has been edited.
$ cat etc/system/local/server.conf
[lmpool:auto_generated_pool_enterprise]
description = auto_generated_pool_enterprise
quota = MAX
slaves = *
stack_id = enterprise

[license]
active_group = Enterprise

[shclustering]

shcluster_label and the secret is not set still in the [shclustering] section. However, when I run the indexer cluster master commands "edit cluster-config" with cluster_label and secret, it is set as expected in the [clustering] section.

0 Karma
Highlighted

Re: Splunk shcluster configuration on master not updating config

Builder

A quick note: splunk help edit shcluster-config does not show shcluster_label in the options - it shows secret and then replication factor param, timeouts and port settings.

Also - I can set labels and secrets fine for the indexer cluster master and cluster. I am trying to also configure the indexer master as the shc deployer, it's the shcluster-config command that is not updating the config. However, I am still able to bring up shc's in a shc cluster with no label specified and it connects / registers to the deployer fine. However, i'd like to specify a secret and a label

0 Karma
Highlighted

Re: Splunk shcluster configuration on master not updating config

Builder
0 Karma
Highlighted

Re: Splunk shcluster configuration on master not updating config

Builder

last post: this doc describes setting up the deployer in step 2 - https://docs.splunk.com/Documentation/Splunk/7.3.1/DistSearch/SHCdeploymentoverview

however, i am looking to use the CLI to configure it, not modifying config files directly.

0 Karma
Highlighted

Re: Splunk shcluster configuration on master not updating config

Path Finder

From my clustering class notes, there is no CLI support for enabling the deployer. You have to edit the server.conf file and then restart splunk:

[shclustering]
pass4SymmKey =

Secret needs to be the same on deployer and SHC members

0 Karma
Highlighted

Re: Splunk shcluster configuration on master not updating config

Builder

thanks. accepting your answer.

0 Karma