Running splunk list forward-server returns "error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca."

[OGS]

Splunk 9.4

Running splunk list forward-server returns "error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca."

I'm forwarding each server's internal logs to the indexer, and I noticed this during a check.
Only the heavy forwarder is returning the error in the title.
Are the configuration methods different for the search head and heavy forwarder?
I don't think there's a problem with the certificate. I think I need to review the settings, but I'm not sure what to do.

The results of running heavy-1 are as follows:
$ sudo /opt/splunk/bin/splunk list forward-server
Couldn't complete HTTP request: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca

Running it on search-head-1 returns the following results:
$ sudo /opt/splunk/bin/splunk list forward-server
Active forwards:
xx.xx.xx.7:9997 (ssl)
Configured but inactive forwards:
None

Manually checking with openssl (Indexer: 9997 and 8089) confirmed that there were no issues with the certificate.
$ sudo openssl s_client -connect xx.xx.xx.7:9997
-CAfile /opt/splunk/etc/auth/mycerts/myCA.pem
-cert /opt/splunk/etc/auth/mycerts/heavy-1.pem
Verify return code: 0 (ok) is returned, indicating successful certificate verification. A Session-ID and Master-Key have been generated, and the session has been established.

The server.conf/[sslConfig] stanza for heavy-1 returns "error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca."
Changing requireClientCert = true to false returns the result.
$ sudo /opt/splunk/bin/splunk list forward-server
Active forwards:
10.2.0.7:9997 (ssl)
Configured but inactive forwards:
None

The following is the configuration for heavy-1. The same configuration is used for search-head-1.
outputs.conf
[tcpout]
defaultGroup = indexer_group

[tcpout:indexer_group]
server = xx.xx.xx.7:9997

disabled = 0
clientCert = /opt/splunk/etc/auth/mycerts/heavy-1.pem
sslVersions = tls1.2
sslCommonNameToCheck = indexer-1,heavy-1,search-head-1
useSSL = true

server.conf
[sslConfig]
enableSplunkdSSL = true
sslRootCAPath = /opt/splunk/etc/auth/mycerts/myCA.pem
serverCert = /opt/splunk/etc/auth/mycerts/heavy-1.pem
requireClientCert = true <= If you set this to false, the splunk list forward-server command will return results.
sslVersions = tls1.2
sslPassword = <pass>
sslCommonNameToCheck = indexer-1,heavy-1,search-head-1
sslVerifyServerCert = true
sslVerifyServerName = true
cliVerifyServerName = true

Below is Indexer
inputs.conf
[splunktcp-ssl:9997]
disabled = 0

[SSL]
serverCert = /opt/splunk/etc/auth/mycerts/indexer-1.pem
sslVersions = tls1.2
requireClientCert = true
sslCommonNameToCheck = heavy-1,search-head-1

server.conf
[sslConfig]
enableSplunkdSSL=true
sslRootCAPath = /opt/splunk/etc/auth/mycerts/myCA.pem
serverCert = /opt/splunk/etc/auth/mycerts/indexer-1.pem
requireClientCert = true
sslVersions = tls1.2
sslPassword = <pass>
sslCommonNameToCheck = indexer-1,heavy-1,search-head-1
sslVerifyServerCert = true
sslVerifyServerName = true
cliVerifyServerName = true

I use Google Translate to translate Japanese into English.

[livehybrid · Accepted solution]

Hi @OGS 

With requireClientCert = true in server.conf,  some of the Splunk CLI commands will not be able to communicate with the Splunk REST port as it is also checked for certificates thus no longer work because it is unable to present a certificate as a client.

https://medium.com/@vikashtalanki/securing-splunk-enterprise-with-ssl-eb2fb568c90e#:~:text=Note%20%3...

It seems that your actual configuration is correct and working, you will just be unable to use the CLI on a server when requireClientCert is set.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[OGS]

Thank you. This information was very useful.

It's a mystery why only Heavy Forwarder was experiencing this issue, but
the workaround solved it.

[httpServerListener:127.0.0.1:<new port>]
ssl=false