Deployment Architecture

Search Head Clustering - Deployer can't contact license master

dflodstrom
Builder

I've recently updated my Splunk lab to begin testing 6.2 and search head clustering. All of my license slaves are communicating successfully with the license master which is still running 6.1.4 with the exception of my deployer.

My deployer functions as a forwarder manager (deployment server), cluster master node, and search head cluster deployer. I still have many servers running 6.1.4 and reporting to the license master. Looking at server.conf on my servers I only see 'pass4SymmKey' in the [general] stanza on the 6.2 systems. The value for 'pass4SymmKey' is different in [general] and [shclustering] and this value existed before I joined the slaves to the master.

On my deployer the value in [general] pass4SymmKey is the same as the value in [shclustering] pass4SymmKey. If i delete this line and restart Splunk it is automatically repopulated under [general]. Is it reading this from another file? How can I fix this?

0 Karma
1 Solution

dflodstrom
Builder

My shclustering pass4SymmKey was not the default value of 'changeme' for [clustering] or [shclustering].

I did change the value of [general] pass4SymmKey to the plain text from /system/default/server.conf and that fixed my issue. There must have been a conflict when I set [general] pass4SymmKey when configuring my Deployer; I didn't realize that this key was used for other functions as well.

In my server.conf now i have different values for pass4SymmKey in [clustering], [general], and [shclustering]. For anyone configuring a Search Head Cluster it should be noted that pass4SymmKey should be set in [shclustering] stanza, especially if you are connecting to a remote license master.

View solution in original post

dflodstrom
Builder

My shclustering pass4SymmKey was not the default value of 'changeme' for [clustering] or [shclustering].

I did change the value of [general] pass4SymmKey to the plain text from /system/default/server.conf and that fixed my issue. There must have been a conflict when I set [general] pass4SymmKey when configuring my Deployer; I didn't realize that this key was used for other functions as well.

In my server.conf now i have different values for pass4SymmKey in [clustering], [general], and [shclustering]. For anyone configuring a Search Head Cluster it should be noted that pass4SymmKey should be set in [shclustering] stanza, especially if you are connecting to a remote license master.

dflodstrom
Builder

My shclustering pass4SymmKey was not the default value of 'changeme' for [clustering] or [shclustering].

I did change the value of [general] pass4SymmKey to the plain text from /system/default/server.conf and that fixed my issue. There must have been a conflict when I set [general] pass4SymmKey when configuring my Deployer; I didn't realize that this key was used for other functions as well.

In my server.conf now i have different values for pass4SymmKey in [clustering], [general], and [shclustering]. For anyone configuring a Search Head Cluster it should be noted that pass4SymmKey should be set in [shclustering] stanza, especially if you are connecting to a remote license master.

0 Karma

arahut_splunk
Splunk Employee
Splunk Employee

the pass4SymmKey in [general] would be used to talk to License Master.

the pass4SymmKey in [general] and clustering would be used to talk to peers.

the pass4SymmKey in [general] and [shclustering] would be used to talk to search heads in the search head cluster.

The pass4SymmKey's would be encrypted, so it will be coming from the default config files into local config files
as the encrypted value. Ignore the fact that it keeps coming back. It is a side-effect of this encryption process.
Your shclustering clear text pass4SymmKey must have been "changeme" which is the same as the
pass4SymmKey in the etc/system/default/server.conf/[general]/pass4SymmKey

For this setup, please ensure that the 3 relevant pass4SymmKey's are exactly as they should be.
If the pass4SymmKey's are different, then you need to use all 3 layers ([general], [clustering] and [shclustering] )

dflodstrom
Builder

If I delete the [general] and [shclustering] stanzas from my server.conf, save my changes, restart splunk, the [general] stanza is repopulated with a hash of the password that matches my [shclustering] pass4SymmKey hash. Maybe we can't havce a deployer operating on a index cluster master.

0 Karma

dflodstrom
Builder

After the upgrade I had no issues contacting the license master. I did not replace any of the configuration files but I did modify server.conf in preparation for search head clustering. There was no 'pass4SymmKey' under the [general] stanza on my master node, I added the key to that stanza for search head clustering. In the docs it says we can use [general] or [shclustering].

I saw the post that you provided in your answer before I opened this one. I attempted to use that to solve my issue but was unsuccessful; it doesn't match what I am experiencing. Also, I have no 'pass4SymmKey' in [general] on any of my 6.1.4 license slaves that are pointed at the same license master.

0 Karma

dflodstrom
Builder
  1. Yes, it is a deployer, cluster master, and deployment server
  2. The key for indexer clustering is different than the key for search head clustering
0 Karma

jayannah
Builder

After upgrade did you replace/edit any configuration files manually? If yes, please describe. Also what are the commands you executed after upgrade?

dflodstrom
Builder

The error I receive when change the master is:

Bad Request — In handler 'localslave': editTracker failed, reason='WARN: path=/masterlm/usage: invalid signature on request from ip=

0 Karma

jayannah
Builder
  1. The same instances configured as Deployer, Cluster Master and Deployment Server - is it right?
  2. Is the key (password) used for search head cluster is also same as key set in cluster master or different?
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...