Why is my Windows Forwarder SSL Configuration not forwarding through?

[shocko]

I'm using Splunk Enterprise 8.2.5 on Windows (both indexers and Forwarders). I have modified inputs.conf on the indexer as follows to referebce my PJI signed certificate/key pair:

[splunktcp-ssl:9998]
disabled = 0

[SSL]
serverCert = C:\Program Files\Splunk\etc\auth\mycert\my.pem
sslPassword = mypassword
requireClientCert = false
sslVersions = *,-ssl2,-ssl3,-tls1.0,-tls1.1

After service restart I see port 9998 listening on the indexer. I added the following config to the outputs.conf of my forwarder:

[tcpout:production]
server = myindexerfqdn:9998
useSSL = true

No data is getting forwarded though and the following is raised in splunkd.log at the forwarder:

03-29-2022 13:01:11.229 +0100 ERROR SSLCommon [37916 parsing] - Can't read certificate file errno=33558528 error:02001000:system library:fopen:system library
03-29-2022 13:01:11.229 +0100 ERROR TcpOutputProc [37916 parsing] - Error initializing SSL context - check splunkd.log regarding configuration error for server myindexerfqdn:9998

What is the windows forwarder looking for? I set the indexer not to verify client certs but does the forwarder need a client certificate (self-signed or otherwise) generated regardless to use SSL ?

[somesoni2]

Your forwarder would need SSL certs and configurations as well to enable SSL communication with your SSL enabled indexer. This documentation will give you all the details: https://docs.splunk.com/Documentation/Splunk/8.2.5/Security/ConfigureSplunkforwardingtousesignedcert...

[shocko]

Since I have told the indexer to ignore client certs what does the client need them for?

[Stefanie]

@shocko 

I had similar problems with my set up for SSL. 

Are you able to run the command:

>openssl.exe rsa -in "C:\Program Files\Splunk\etc\auth\mycert\my.pem" -text

Try following the steps listed here if you haven't

https://docs.splunk.com/Documentation/Splunk/8.2.5/Security/Troubleshootyouforwardertoindexerauthent...

[shocko]

So I have verified the indexer is listening on TCP 9998 and has my PKI certificate bound to it. I don't understand why the forwarder needs any client certificates to use SSL. I just want to ensure the data is forwarded over SSL. Why would I need a client certificate for this? I'm using deployment server to deploy my apps so I also don't want to specify the password for the client certificate private key in the outputs.conf. 

[Stefanie]

---

@shocko wrote:

So I have verified the indexer is listening on TCP 9998 and has my PKI certificate bound to it. I don't understand why the forwarder needs any client certificates to use SSL. I just want to ensure the data is forwarded over SSL. Why would I need a client certificate for this? 

---

For the SSL connection to the indexers the forwarder requires a certificate. The clientCert is used to "turn on" SSL connections. That's my assumption. 

You can use the certificate you created for your indexers to use on your forwarders.

---

@shocko wrote:

I'm using deployment server to deploy my apps so I also don't want to specify the password for the client certificate private key in the outputs.conf. 

---

Splunk doesn't support setting up SSL certificates in apps for this very reason anymore. It took me a long time of trial and error before someone @ Splunk told me this. You'll need to place your certificate somewhere in $SPLUNK_HOME/etc/auth/(folder) and your outputs.conf in $SPLUNK_HOME/etc/system/local

Touching back on your error you received on the splunkd.log on your forwarder, if you restart the indexer do you see where your indexer is successfully accepting SSL? It might say something like 

port 9998 is reserved for splunk 2 splunk (SSL)

[shocko · Accepted solution]

So I resolved my specific issue as follows:

Since my indexer is using a PKI signed certificate and that PKI has a Root CA and Issuing CA I had to add the Issuing CA public cert and Root CA to a .PEM file (in that order) and drop onto my forwarder

In outputs.conf I then reference it as follows:

[tcpout:test-ssl-1]
disabled = 0
server = indexer1.mydomain.com:9998
useSSL = true
useClientSSLCompression = true

sslVerifyServerCert = false
sslRootCAPath = C:\Program Files\SplunkUniversalForwarder\etc\auth\CA_Chain.pem

So I have a working setup with the indexer using a PKI signed certificate and the forwarder without defining any client certs. Even though sslVerifyServerCert is set to false I still need to supply sslRootPath. Again, I don't know why as it doesn't make sense to me 😐

My takeaways:

1. In order for the forwarder to ship events to the indexer over SSL a client certificate does not need be defined on the forwarder outputs.conf files
2. The statement regarding the password for the client PEM file not being encrypted if it's defined n inputs.conf or outputs.conf outside of /etc/system/local/ does not appear to be true in 8.2.5 as my passwords are getting encrypted in those config files under the apps directory when the forwarder restarts
3. If you wish to verify the indexer cert and it is using a PKI then you must point the forwarder at a PEM file that contains all CAs in that chain from bottom to top