Search Head Clustering - Deployer can't contact license master

[dflodstrom]

I've recently updated my Splunk lab to begin testing 6.2 and search head clustering. All of my license slaves are communicating successfully with the license master which is still running 6.1.4 with the exception of my deployer.

My deployer functions as a forwarder manager (deployment server), cluster master node, and search head cluster deployer. I still have many servers running 6.1.4 and reporting to the license master. Looking at server.conf on my servers I only see 'pass4SymmKey' in the [general] stanza on the 6.2 systems. The value for 'pass4SymmKey' is different in [general] and [shclustering] and this value existed before I joined the slaves to the master.

On my deployer the value in [general] pass4SymmKey is the same as the value in [shclustering] pass4SymmKey. If i delete this line and restart Splunk it is automatically repopulated under [general]. Is it reading this from another file? How can I fix this?

[dflodstrom · Accepted solution]

My shclustering pass4SymmKey was not the default value of 'changeme' for [clustering] or [shclustering].

I did change the value of [general] pass4SymmKey to the plain text from /system/default/server.conf and that fixed my issue. There must have been a conflict when I set [general] pass4SymmKey when configuring my Deployer; I didn't realize that this key was used for other functions as well.

In my server.conf now i have different values for pass4SymmKey in [clustering], [general], and [shclustering]. For anyone configuring a Search Head Cluster it should be noted that pass4SymmKey should be set in [shclustering] stanza, especially if you are connecting to a remote license master.

[dflodstrom]

My shclustering pass4SymmKey was not the default value of 'changeme' for [clustering] or [shclustering].

I did change the value of [general] pass4SymmKey to the plain text from /system/default/server.conf and that fixed my issue. There must have been a conflict when I set [general] pass4SymmKey when configuring my Deployer; I didn't realize that this key was used for other functions as well.

In my server.conf now i have different values for pass4SymmKey in [clustering], [general], and [shclustering]. For anyone configuring a Search Head Cluster it should be noted that pass4SymmKey should be set in [shclustering] stanza, especially if you are connecting to a remote license master.

[arahut_splunk]

the pass4SymmKey in [general] would be used to talk to License Master.

the pass4SymmKey in [general] and clustering would be used to talk to peers.

the pass4SymmKey in [general] and [shclustering] would be used to talk to search heads in the search head cluster.

The pass4SymmKey's would be encrypted, so it will be coming from the default config files into local config files
as the encrypted value. Ignore the fact that it keeps coming back. It is a side-effect of this encryption process.
Your shclustering clear text pass4SymmKey must have been "changeme" which is the same as the
pass4SymmKey in the etc/system/default/server.conf/[general]/pass4SymmKey

For this setup, please ensure that the 3 relevant pass4SymmKey's are exactly as they should be.
If the pass4SymmKey's are different, then you need to use all 3 layers ([general], [clustering] and [shclustering] )

[dflodstrom]

If I delete the [general] and [shclustering] stanzas from my server.conf, save my changes, restart splunk, the [general] stanza is repopulated with a hash of the password that matches my [shclustering] pass4SymmKey hash. Maybe we can't havce a deployer operating on a index cluster master.

[dflodstrom]

After the upgrade I had no issues contacting the license master. I did not replace any of the configuration files but I did modify server.conf in preparation for search head clustering. There was no 'pass4SymmKey' under the [general] stanza on my master node, I added the key to that stanza for search head clustering. In the docs it says we can use [general] or [shclustering].

I saw the post that you provided in your answer before I opened this one. I attempted to use that to solve my issue but was unsuccessful; it doesn't match what I am experiencing. Also, I have no 'pass4SymmKey' in [general] on any of my 6.1.4 license slaves that are pointed at the same license master.

[jayannah]

Try .. http://answers.splunk.com/answers/188557/620-to-61x-license-master-issues.html
http://answers.splunk.com/answers/187208/my-62-license-slave-is-not-working-my-614-license.html

[dflodstrom]

1. Yes, it is a deployer, cluster master, and deployment server
2. The key for indexer clustering is different than the key for search head clustering

[jayannah]

After upgrade did you replace/edit any configuration files manually? If yes, please describe. Also what are the commands you executed after upgrade?

[dflodstrom]

The error I receive when change the master is:

Bad Request — In handler 'localslave': editTracker failed, reason='WARN: path=/masterlm/usage: invalid signature on request from ip=

[jayannah]

1. The same instances configured as Deployer, Cluster Master and Deployment Server - is it right?
2. Is the key (password) used for search head cluster is also same as key set in cluster master or different?