Is there a method to change the pass4SymmKey value across all the peers in an index cluster and also a search-head cluster to ensure replication within the index cluster and integration of the search-head cluster with an indexer cluster?

[transtrophe]

I tried doing the following in an effort to troubleshoot and resolve an issue I am having with an initial distributed deployment of Splunk:

On the cluster master I created a server.conf in the cluster bundle (/opt/splunk/etc/master-apps/_cluster/local/) with only the clustering stanza and a single entry for the updated pass4SymmKey:

[clustering]
pass4SymmKey = myNewKeyPass

I did the same thing on the search-head cluster deployer's bundle.

I then applied both bundles. On the indexers the "pushed" server.conf was successfully deployed to /opt/splunk/etc/slave-apps/_cluster/local and on the search-heads to opt/splunk/etc/apps/StandaloneConfigs/default/. [Note: I am using the sub-dir titled StandAloneConfigs to push configuration files from the shc deployer to the shc members.]

I then did a rolling restart of both clusters. The index cluster failed to establish so I checked the values of the server.conf on the indexers using btool and saw that the value for pass4SymmKey in each indexers merged server.conf never got hashed (it was the plain-text passphrase). Also, the values were sourced from /opt/splunk/etc/slave-apps/_cluster/local/server.conf files. On the search-head cluster members, the "pushed" server.conf that deployed to opt/splunk/etc/apps/StandaloneConfigs/default/ did NOT get used in the merge but the initial /opt/splunk/etc/system/local/server.conf got used.

So if anyone has a correct/best-practices method for doing this kind of large-scale pass4SymmKey update to accomplish the combined objectives of 1.) ensuring coordination within an index cluster and 2.) ensuring coordination between a search=head cluster and an indexer cluster for forwarding the search-head cluster _internal events to the designated indexer cluster layer, that would be awesome.

[esix_splunk · Accepted solution]

For clustering, do not configure the actual cluster settings as an APP in master-apps on the Cluster Master. Why, because the members have to be joined to this first, so you have to set this locally and then you're going to have potential for configuration collisions and other issues. If you want to update this on a large scale, your best bet would be to use a combination of ssh and sed/awk on the local server.conf to replace the pass4Symmkey.

For cluster functionality, set it on each peer in $splunk_home$/etc/system/local.

Use master-apps to deploy index and app configurations to your indexers.

[transtrophe]

I would like to recommend that Splunk consider building in this functionality to the Enterprise core capabilities - a kind of PKI mechanism, really. There are several situations I can envision that would call for this kind of large-scale deployment of new key material across complex indexer and search-head deployments. Say a large deployment experienced a hack. In that situation, best-practices from a GRC perspective would call for re-issuing key material. LIkewise, several industries such as the nuclear power generating infrastructures (for NEI 08-09 / 10 CFR 73.54) and the bulk electric system infrastructures (for NERC CIP) have password and secure information establishment requirements from a regulatory perspective that would call for this kind of key management capabilities.

This could be done as you suggest with some set of home-grown approaches, but Splunk provided mechanisms would ensure a far greater degree of deployment and maintenance consistency.

[buysse]

transtrophe - I'm late to this party, but did you ever file this as an enhancement request? I'll happily pile on if you did.