Deployment Architecture

Splunk shcluster configuration on master not updating config

brettcave
Builder

I am trying to configure a master node in a splunk cluster to be an indexer master and shc deployer. I install Splunk Enterprise (7.3.1) and then execute commands to configure it:

splunk edit shcluster-config -secret clustersecret -shcluster_label myshclabel -auth user:pass

After doing this, etc/system/local/server.conf is updated with a [shcluster] heading, but no pass4SymmKey or cluster label is configured. it is a blank stanza.

I am also running edit cluster-config on the server and this is updating the config as expected. My understanding is that these commands should add a cluster label and secret to the config. Am I missing something? Not sure if licensing needs to be configured first? For my use-case, manually editing the file is not an option, it needs to be configured by command.

Tags (2)
0 Karma
1 Solution

jdhunter
Path Finder

Do you have a License installed on this server? I would setup a License Master and point all of your Splunk instances to it.

Once you have a license master configured, you can run this command on all your Splunk servers.

Setting License Master Slave

splunk edit licenser-localslave -master_uri 'https://license_master_uri:8089'

https://docs.splunk.com/Documentation/Splunk/7.3.1/Admin/Configurealicensemaster
https://docs.splunk.com/Documentation/Splunk/7.3.1/Admin/LicenserCLIcommands

Next, I would setup the Cluster Master and the Cluster Peers

Cluster Master

splunk edit cluster-config -mode master -replication_factor # -search_factor # -secret your_key -cluster_label cluster1

Cluster Peer (Your Indexers)

splunk edit cluster-config -mode slave -master_uri https://cluster_master:8089 -replication_port 9887 -secret your_key

https://docs.splunk.com/Documentation/Splunk/7.3.1/Indexer/Enablethemasternode
https://docs.splunk.com/Documentation/Splunk/7.3.1/Indexer/ConfiguremasterwithCLI
https://docs.splunk.com/Documentation/Splunk/7.3.1/Indexer/ConfigurepeerswithCLI

Finally, there are multiple steps in setting up a Search Head Cluster to Deploy the cluster:

These are the key steps in deploying clusters:

  1. Identify your requirements.

  2. Set up the deployer.

  3. Install the Splunk Enterprise instances.

  4. Initialize cluster members.

  5. Bring up the cluster captain.

  6. Perform post-deployment set-up.

https://docs.splunk.com/Documentation/Splunk/7.3.1/DistSearch/SHCdeploymentoverview

System Requirements for installing Cluster Master and SHC Deployer on the same machine

https://docs.splunk.com/Documentation/Splunk/7.3.1/Indexer/Systemrequirements

View solution in original post

jdhunter
Path Finder

Do you have a License installed on this server? I would setup a License Master and point all of your Splunk instances to it.

Once you have a license master configured, you can run this command on all your Splunk servers.

Setting License Master Slave

splunk edit licenser-localslave -master_uri 'https://license_master_uri:8089'

https://docs.splunk.com/Documentation/Splunk/7.3.1/Admin/Configurealicensemaster
https://docs.splunk.com/Documentation/Splunk/7.3.1/Admin/LicenserCLIcommands

Next, I would setup the Cluster Master and the Cluster Peers

Cluster Master

splunk edit cluster-config -mode master -replication_factor # -search_factor # -secret your_key -cluster_label cluster1

Cluster Peer (Your Indexers)

splunk edit cluster-config -mode slave -master_uri https://cluster_master:8089 -replication_port 9887 -secret your_key

https://docs.splunk.com/Documentation/Splunk/7.3.1/Indexer/Enablethemasternode
https://docs.splunk.com/Documentation/Splunk/7.3.1/Indexer/ConfiguremasterwithCLI
https://docs.splunk.com/Documentation/Splunk/7.3.1/Indexer/ConfigurepeerswithCLI

Finally, there are multiple steps in setting up a Search Head Cluster to Deploy the cluster:

These are the key steps in deploying clusters:

  1. Identify your requirements.

  2. Set up the deployer.

  3. Install the Splunk Enterprise instances.

  4. Initialize cluster members.

  5. Bring up the cluster captain.

  6. Perform post-deployment set-up.

https://docs.splunk.com/Documentation/Splunk/7.3.1/DistSearch/SHCdeploymentoverview

System Requirements for installing Cluster Master and SHC Deployer on the same machine

https://docs.splunk.com/Documentation/Splunk/7.3.1/Indexer/Systemrequirements

brettcave
Builder

@jdhunter - thanks for bringing up the license question - I was not installing the license. I have updated it to do so, but am still not seeing the right config get added.

$ splunk add licenses /path/to/enterprise.lic 
The licenses object has been added
You need to restart the Splunk Server (splunkd) for your changes to take effect.
$ service splunk restart
$ splunk list licenses # enterprise license shows.
$ splunk list licenser-pools
    auto_generated_pool_enterprise
        description:auto_generated_pool_enterprise
        effective_quota:REDACTED
        is_unlimited:0
        quota:MAX
        slaves:
        stack_id:enterprise
        used_bytes:0
$ splunk edit shcluster-config -secret testingsecret  -shcluster_label shclusterlabel_test
The shcluster-config property has been edited.
$ cat etc/system/local/server.conf
[lmpool:auto_generated_pool_enterprise]
description = auto_generated_pool_enterprise
quota = MAX
slaves = *
stack_id = enterprise

[license]
active_group = Enterprise

[shclustering]

shcluster_label and the secret is not set still in the [shclustering] section. However, when I run the indexer cluster master commands "edit cluster-config" with cluster_label and secret, it is set as expected in the [clustering] section.

0 Karma

brettcave
Builder

A quick note: splunk help edit shcluster-config does not show shcluster_label in the options - it shows secret and then replication factor param, timeouts and port settings.

Also - I can set labels and secrets fine for the indexer cluster master and cluster. I am trying to also configure the indexer master as the shc deployer, it's the shcluster-config command that is not updating the config. However, I am still able to bring up shc's in a shc cluster with no label specified and it connects / registers to the deployer fine. However, i'd like to specify a secret and a label

0 Karma

brettcave
Builder
0 Karma

brettcave
Builder

last post: this doc describes setting up the deployer in step 2 - https://docs.splunk.com/Documentation/Splunk/7.3.1/DistSearch/SHCdeploymentoverview

however, i am looking to use the CLI to configure it, not modifying config files directly.

0 Karma

jdhunter
Path Finder

From my clustering class notes, there is no CLI support for enabling the deployer. You have to edit the server.conf file and then restart splunk:

[shclustering]
pass4SymmKey =

Secret needs to be the same on deployer and SHC members

0 Karma

brettcave
Builder

thanks. accepting your answer.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...