How to Set up TLS Between a Universal Forwarder (UF) and an Indexer

[tt-nexteng]

1.

This is $SPLUNK_HOME/etc/system/local/inputs.conf of my Indexer.

[splunktcp-ssl:9997]
disabled = 0

[SSL]
serverCert = /opt/splunk/etc/auth/mycerts/myCombinedServerCertificate.pem
sslPassword = $7$2sKE3fmGeaZOyBdYg6AfpoU1Gv7kXP3pEihEQoWlSKFeItPCn0lNyb0=  (myServerPrivateKeyPassword)
requireClientCert = false

2.

This is $SPLUNK_HOME/etc/system/local/server.conf of my Indexer.

[general]
serverName = 4b2c00e08e88
pass4SymmKey = $7$kbQmQuYtD+ees5uv8q+WaE36j8Sk07HcWoVgOMmP8Bb69nbwERriow==

[sslConfig]
sslPassword = $7$9eO6Wt/mPl2QIOEu/+xh44foXzSDvMRs/0LyNn/EuZ+ab/Q93LB8bg==(Default. I did not modify)
sslRootCAPath = /opt/splunk/etc/auth/mycerts/myCertAuthCertificate.pem

[lmpool:auto_generated_pool_download-trial]
description = auto_generated_pool_download-trial
peers = *
quota = MAX
stack_id = download-trial

[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
peers = *
quota = MAX
stack_id = forwarder

[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
peers = *
quota = MAX
stack_id = free

3.

This is /opt/splunkforwarder/etc/system/local/outputs.conf of my UF.

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = 176.32.83.56:9997
disabled = 0
sslPassword = $7$M4MRBHX8rh11KC509o7cRe/QOxo3EZBA5pXjGn5cZuHtb0FO3dFj5ks=(myServerPrivateKeyPassword)
sslVerifyServerCert = false

4.

This is /opt/splunkforwarder/etc/system/local/server.conf of my UF.

[general]
serverName = suf
pass4SymmKey = $7$hE1rQcMJG9ZPB0DvxG+KMGbMmNly4JylVUhFC59Nz+LBa+o1ahblmg==

[sslConfig]
sslPassword = $7$30hXe/EpmNqvXRzVPC0KF+1YNptHuhrxEnChvX5Se8ySRni+uAQFHWk=(Default. I did not modify)
sslRootCAPath = /opt/splunkforwarder/etc/auth/mycerts/myCertAuthCertificate.pem

[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
peers = *
quota = MAX
stack_id = forwarder

[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
peers = *
quota = MAX
stack_id = free

The above configurations did not succeed. When I executed the following command:

/opt/splunkforwarder/bin/splunk list forward-server

I got the following result.

Active forwards:
None
Configured but inactive forwards:
176.32.83.56:9997

[tt-nexteng · Accepted solution]

I have resolved this issue. The cause was in the UF outputs.conf configuration.
Thank you all for your help.

However, I don't understand why this configuration is required.
I have posted a new question.

https://community.splunk.com/t5/Security/Questions-about-UF-outputs-conf-Configuration/m-p/710701#M1...

[moliminous]

Your config looks good, so it could be the certs were not prepared correctly, or Splunk cannot read them.
Splunk's docs are not clear or accurate for cert prep.

The server cert on the Indexer must have the leaf cert followed by the private key and that's it.
Any intermediate or root certs are simply referenced by the sslRootCAPath in server.conf

Ensure those cert files are all readable and owned by the 'splunk' user and chmod them to 640 to be safe.
Make sure you can cat the cert and root using the the splunk user on the indexer.

By the way for log encryption, Splunk only uses the server cert (Indexer cert) to encrypt the logs.

As others mentioned, use the openssl command and check the cert results from it.

$SPLUNK_HOME/bin/splunk cmd openssl s_client -connect <your_indexer>:<port> -showcerts

 Also search the internal logs on both the indexer and the UF for TLS errors.

cat /opt/splunk/var/log/splunk/splunkd.log | grep -i 'tls\|ssl'

[tt-nexteng]

The erros from UF log.

02-04-2025 03:53:19.270 +0000 WARN SSLOptions [0 MainThread] - server.conf/[sslConfig]/sslVerifyServerCert is false disabling certificate validation; must be set to "true" for increased security
02-04-2025 03:53:19.274 +0000 ERROR SSLCommon [0 MainThread] - Can't read key file /opt/splunkforwarder/etc/auth/server.pem SSL error code=101077092 message="error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt"
02-04-2025 03:53:19.274 +0000 ERROR ServerConfig [0 MainThread] - Couldn't initialize SSL Context for HTTPClient in ServerConfig
02-04-2025 03:53:19.274 +0000 INFO ServerConfig [0 MainThread] - disableSSLShutdown=0
02-04-2025 03:53:19.658 +0000 INFO ProxyConfig [13255 MainThread] - Successfully initialized enable_tls_proxy=0 from server.conf for splunkd.
02-04-2025 03:53:19.658 +0000 INFO loader [13255 MainThread] - TLS proxy is not enabled. Will not start the TLS proxy server.
02-04-2025 03:53:19.798 +0000 INFO TcpOutputProc [13334 parsing] - Initializing connection for non-ssl forwarding to 176.32.83.56:9997
02-04-2025 03:53:19.886 +0000 INFO loader [13291 HTTPDispatch] - Setting SSL configuration.
02-04-2025 03:53:19.886 +0000 INFO loader [13291 HTTPDispatch] - Server supporting SSL versions=TLS1.2
02-04-2025 03:53:19.898 +0000 ERROR HTTPServer [13291 HTTPDispatch] - Error setting up TLS, TLS will not be enabled. file=server Exception: Can't read key file /opt/splunkforwarder/etc/auth/server.pem SSL error code=101077092 message="error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt"

[moliminous]

On the IDX's server.conf you need to add this line in the [sslConfig] stanza:

serverCert = /opt/splunk/etc/auth/mycerts/myCombinedServerCertificate.pem

Then delete the sslPassword line from your server.conf if it's the default, Splunk will recreate it anyhow.

That should fix it unless your cert is not prepared properly with just leaf cert + private key in the 'myCombinedServerCertificate.pem'

__PRESENT

__PRESENT

[kiran_panchavat]

@tt-nexteng 

The password to the RSA private key that is in the server certificate file is wrong. This generates an error similar to the following: Can't read key file /opt/splunkforwarder/etc/auth/server.pem

02-04-2025 03:53:19.898 +0000 ERROR HTTPServer [13291 HTTPDispatch] - Error setting up TLS, TLS will not be enabled. file=server Exception: Can't read key file /opt/splunkforwarder/etc/auth/server.pem SSL error code=101077092 message="error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt"

[tt-nexteng]

Errors from indexer.

02-04-2025 08:56:10.854 +0000 ERROR TcpInputProc [9178 FwdDataReceiverThread-0] - Error encountered for connection from src=57.180.248.191:52696. error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol

[PickleRick]

OK. "Did not succeed" doesn't tell us much.

1. What do the logs say (on both ends)? They should tell you if the connection has been attempted, if it failed, how it failed and so on.

2. Try connecting directly from the UF machine to the indexer using openssl s_client

splunk cmd openssl s_client -connect <your_indexer>:<port> -showcerts

And see if it works.

[tt-nexteng]

I executed the following command on the Indexer:

[root@f54fbdb70e7d mycerts]# $SPLUNK_HOME/bin/splunk cmd openssl verify -CAfile /opt/splunk/etc/auth/mycerts/myCertAuthCertificate.pem /opt/splunk/etc/auth/mycerts/myCombinedServerCertificate.pem
/opt/splunk/etc/auth/mycerts/myCombinedServerCertificate.pem: OK

Then, I copied the contents of
/opt/splunk/etc/auth/mycerts/myCertAuthCertificate.pem
to
/opt/splunkforwarder/etc/auth/mycerts/myCertAuthCertificate.pem on the Universal Forwarder (UF).

Why am I still getting the error "Verify return code: 19 (self signed certificate in certificate chain)"?

[tt-nexteng]

I restarted the Universal Forwarder (UF) and will now post the current log.

UF

02-04-2025 09:39:14.415 +0000 WARN  SSLOptions [0 MainThread] - server.conf/[sslConfig]/sslVerifyServerCert is false disabling certificate validation; must be set to "true" for increased security
02-04-2025 09:39:14.419 +0000 INFO  ServerConfig [0 MainThread] - SSL session cache path  enabled 0 session timeout on SSL server 300.000
02-04-2025 09:39:14.419 +0000 INFO  ServerConfig [0 MainThread] - disableSSLShutdown=0
02-04-2025 09:39:14.803 +0000 INFO  ProxyConfig [16169 MainThread] - Successfully initialized enable_tls_proxy=0 from server.conf for splunkd.
02-04-2025 09:39:14.803 +0000 INFO  loader [16169 MainThread] - TLS proxy is not enabled. Will not start the TLS proxy server.
02-04-2025 09:39:14.943 +0000 INFO  TcpOutputProc [16248 parsing] - Initializing connection for non-ssl forwarding to 176.32.83.56:9997
02-04-2025 09:39:15.035 +0000 INFO  loader [16205 HTTPDispatch] - Setting SSL configuration.
02-04-2025 09:39:15.035 +0000 INFO  loader [16205 HTTPDispatch] - Server supporting SSL versions=TLS1.2

Indexer

02-04-2025 09:39:15.098 +0000 ERROR TcpInputProc [9178 FwdDataReceiverThread-0] - Error encountered for connection from src=57.180.248.191:34536. error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol

[kiran_panchavat]

@tt-nexteng 

• Examine the splunkd.log file on both the Universal Forwarder and the Indexer for any TLS-related error messages. This can provide clues about what might be going wrong.
• Review your Docker configuration to ensure that all necessary ports are exposed and that the container has the correct network settings.

https://docs.splunk.com/Documentation/Splunk/9.4.0/Security/Validateyourconfiguration

[kiran_panchavat]

@tt-nexteng 

Check this video for reference https://www.youtube.com/watch?v=vI7466EwG7I

[kiran_panchavat]

@tt-nextengCheck this documentation Configure Splunk indexing and forwarding to use TLS certificates - Splunk Documentation

[tt-nexteng]

I have tried many times following this document, but I keep failing.

If I remove the TLS settings and revert these four configuration files to their original state before modification, everything works fine.

Additionally, I am running the indexer in Docker. Could this be related to the issue?