How to make the deployment server manage all Universal Forwarders' server.conf account for system unique fields like "sslKeysfilePassword ” and “pass4SymmKey”?

[EdgarAllenProse]

The goal is to have the deployment server manage server.conf on all Universal Forwarders, like it does with inputs/outputs.conf. Automation is preferred as there are over 300 Windows systems.

E.g. When we make certificate updates, change the sslVersions, and/or the allowed cipherSuite, we want the deployment server to handle it all.

This is an issue as the server.conf includes four fields that appear to be unique to each system, and based on our understanding the deployment server updates the whole file, not per stanza:

• sslKeysfilePassword
• sslPassword
• pass4SymmKey
• serverName

How do deployment servers handle system unique fields so the deployment server doesn’t just overwrite them and cause configuration issues? Any tips for what direction I need to look in? I would appreciate any help as manually updating all universal forwarders would be insanely time consuming.

Here is a scrubbed version of the relevant fields for our deployment server's ~/default/server.conf:

[sslConfig]
enableSplunkdSSL = true
useClientSSLCompression = true
useSplunkdClientSSLCompression = true
# enableSplunkSearchSSL has been moved to web.conf/[settings]/enableSplunkWebSSL

#Allow only sslv3 and above connections to the HTTP server
sslVersions = *,-ssl2
sslVersionsForClient = *,-ssl2
sendStrictTransportSecurityHeader = false
allowSslCompression = true
allowSslRenegotiation = true

# For the HTTP server, Diable ciphers lower than 128-bit and disallow ciphers that
# don't provide authentication and/or encryption.
# Use 'openssl ciphers -v' to generate a list of supported ciphers
# Allow only TLSv1 cipher with 'high' encryption suits, i.e. whose key lengths are
# larger than or equal to 128 bits
cipherSuite = TLSv#+HIGH:TLSv#.2+HIGH:@STRENGTH

serverCert = $SPLUNK_HOME/etc/auth/server.pem
sslPassword = ######
caCertFile = $SPLUNK_HOME/etc/auth/cacert.pem
certCreateScript = $SPLUNK_HOME/bin/splunk, createssl, server-cert
# DEPRECATED
caPath = $SPLUNK_HOME/etc/auth

[applicationsManagement]
updateTimeout = #h
sslVersions = tls#.#
caCertFile = $SPLUNK_HOME/etc/auth/#####.pem
sslVerifyServerCert = true
sslCommonNameToCheck = apps.splunk.com, cdn.apps.splunk.com
sslAltNameToCheck = splunkbase.splunk.com, apps.splunk.com, cdn.apps.splunk.com
cipherSuite = TLSv#+HIGH:@STRENGTH

[clustering]
mode = disabled
pass4SymmKey =
register_replication_address =
register_forwarder_address =
register_search_address =
executor_workers = 10
manual_detention = false

encrypt_fields = "server: :sslKeysfilePassword", "server: :sslPassword", "server: :pass4SymmKey", "server: :password", "outputs:tcpout:sslPassword", "outputs:indexer_discovery:pass4SymmKey", "inputs:SSL:password$

Here is a scrubbed version belonging to one of the windows systems:

[general]
serverName = <Server Name>
pass4SymmKey = $1$###############

[sslConfig]
sslKeysfilePassword = $###############

[gjanders · Accepted solution]

Splunk configuration file precedence should help explain this concept. I have all the configuration that needs overriding, including that of the server.conf file in an application which drops into $SPLUNK_HOME/etc/apps//...

You only want to override in the local directory, not the default directory, so just override the fields that you wish to override, for example:

[sslConfig]
sslVersions = tls1.2

[general]
site = site0

Or similar, you only need to override the parameters that you want overriden. If you wanted to override the pass4SymmKey/sslKeysfilePassword then putting that in an app would work fine unless you have created system/local versions (as per the precedence page linked above).