SSL handshake failure between Universal Forwarder and Indexer

[JahanviVV]

I am facing an SSL handshake issue after renewing the certificate on our indexer node.

Earlier data ingestion from the UF server stopped after certificate renewal.
When we revert to the old certificate and set `sslVerifyServerCert = false` in outputs.conf, data flows successfully.

With the new certificate in place, SSL handshake fails with:
"ssl23_write:ssl handshake failure" and "no peer certificate available" errors.

Indexer and UF details:
- Indexer
- UF
- Port: 9997 (SSL enabled)
- Splunk UF Version: 9.1.7
- Splunk Enterprise Version: 7.2.3
- Certificate renewed from internal CA (PepsiCoCA01)

Please assist in identifying the issue

[livehybrid]

Hi @JahanviVV 

Is it the same CA as previously used? Are there any subCA between the new server cert and the CA?

You could use the following openssl command to check the cert presented by your Indexer is as expected:

openssl s_client -connect <indexer-FQDN>:9997 -showcerts

This command connects to the indexer's port 9997 with SSL, retrieves and displays the entire certificate chain presented by the indexer. The PEM-encoded certificate section output can be reviewed (CN/SAN, issuer, validity dates). The certificate at the top is the one directly presented by the indexer.

You can also try adding -CAfile <pathToCA_on_UF> to the command with which you should get a Verify return code: 0 (ok).

As @richgalloway said - if you are using 7.2.3 then this is well out of support and its not beyond the realm of possibility that this is causing an issue.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[JahanviVV]

Hi,
Thank you for your response. Yes, the new certificate is issued by the same internal CA (PepsiCoCA01). The new certificate includes both Root CA and Intermediate CA in the chain. When I run the command: openssl s_client -connect ppsplix01.corp.pep.pvt:9997 -showcerts
I get the error: "no peer certificate available" and "ssl handshake failure". This confirms that the indexer is not presenting any certificate. We suspect that Splunk 7.2.3 may not be able to properly load or handle the renewed certificate, as the environment still uses this version (due for upgrade).
Could you please confirm if this issue is related to the old OpenSSL version in Splunk 7.2.3, or if there is any workaround to make it work temporarily until we upgrade?
Thank you.

[richgalloway]

Is there a typo in the message or are you really running Splunk 7?  If you are then that's a likely source of the problem.  Try upgrading to a supported version of Splunk.