Difference between pass4SymmKey and SSL in a distributed environment, and what to use for indexing?

[aamer86]

Hi,

Can someone help me understand the difference between pass4symmkey and SSL settings for secure Splunk connections in a distributed environment?

What should we use for indexing? Cluster peers communications?

[mguhad]

The pass4SymKey is splunk's special key to let your indexers or other relevent instances (SHs, CM<>IDX) communite and authenticate with each other. In short, pass4Symkey controls the authentication between splunk instances and does not control user access. see here for more infro : https://docs.splunk.com/Documentation/Splunk/8.0.3/Security/Aboutsecuringclusters.

SSL is related to the encryption of comms/data between splunk instances and is unrelated to the functionality of the pass4Symkey as you can configure indexing of data without SSL (its optional, but highly recommended step for hardenening your env against casual snoopers etc..).

For indexing a cluster peer communications, SSL is optional. But you MUST use pass4symkey otherwise the cluster peers will not be able to join the cluster and authenticate against the cluster master. You do this by setting the security key* (-secret) on the cluster peers which is then converted to pass4Symkey (server.conf).
Simply configure the peers to have same -secret as Cluster master and then open the 9997 port on the indexers (inputs.conf as an app from CM) and you should be good to go in terms of indexing data!

*Security key - This is the key that authenticates communication between the master and the peers and search heads. The key must be the same across all cluster nodes. Set the same value here that you previously set on the master node.
More reading here: https://docs.splunk.com/Documentation/Splunk/8.0.3/Indexer/Enablethepeernodes

Hope this helps!

[aamer86]

thanks @mguhad

Does that mean we dont need to configure it on universal forwarders , as i see it in the server.conf in the installed UF.

and if leave the default value it takes on the UF will that impact its connections to the DS/Indexing cluster if the value is different on them

[mguhad]

@aamer86 Yes, the defaults should work fine. You generally wouldnt want to fiddle with parameter that unless you are doing clustering (only applicable to indexers adn Cluster master) or SSL encryption (applicable to UFs too). It won't affect your clusters indexing/DS unless you have configured SSL and have set "requireClientCert = true" on the indexers etc. The values will always be differnt because of how splunk Hashes the plaintext password - passwords from sslpassword, -secret from clustering etc.

But if you are not configuring ssl, you just need to configure outputs.conf on the UFs to send data to the indexers' recieving port (inputs.conf on indexers.)

Please accept this answer if it has helped so others can benefit. Thanks.

[aamer86]

I need to ensure sending logs securely
so I will use certificates on both Linux and windows servers for SSL but what confused me is the Pass4symmkey which I am aware we need to set and save in the indexing cluster so if we add any more peers in the future we have it.

[mguhad]

@aamer86 ,
Yes well then in that case you will need to keep an eye on the pass4symkey (it basically MASKS plaintext passwords (i.e from -secret key). Once you have new members joining the cluster/env...simplly comment out their current pass4symkey/sslpassword value, then push the new value out ina app - you will notice splunk will automatically regenerate that value and encrypt that plain text password) . Pass4Symkey and sslpassword are two completely different things to keep an eye on when using custom SSL certs & clustering.

Just ensure that all nodes can have access to the correct SSLpassword parameter (this is also generated automatically by splunk using its default certs, if you custom ones, simple comment this out, apply your custom certs and splunk will generate new sslpassword based on what you have just configured like above for pas4symkey.)

More on that can be found here: https://docs.splunk.com/Documentation/Splunk/8.0.4/Security/WhatyoucansecurewithSplunk