Getting Data In

Why is my Windows Forwarder SSL Configuration not forwarding through?

shocko
Contributor

I'm using Splunk Enterprise 8.2.5 on Windows (both indexers and Forwarders). I have modified inputs.conf on the indexer as follows to referebce my PJI signed certificate/key pair:

[splunktcp-ssl:9998]
disabled = 0

[SSL]
serverCert = C:\Program Files\Splunk\etc\auth\mycert\my.pem
sslPassword = mypassword
requireClientCert = false
sslVersions = *,-ssl2,-ssl3,-tls1.0,-tls1.1

After service restart I see port 9998 listening on the indexer. I added the following config to the outputs.conf of my forwarder:

[tcpout:production]
server = myindexerfqdn:9998
useSSL = true

No data is getting forwarded though and the following is raised in splunkd.log at the forwarder:

03-29-2022 13:01:11.229 +0100 ERROR SSLCommon [37916 parsing] - Can't read certificate file errno=33558528 error:02001000:system library:fopen:system library
03-29-2022 13:01:11.229 +0100 ERROR TcpOutputProc [37916 parsing] - Error initializing SSL context - check splunkd.log regarding configuration error for server myindexerfqdn:9998

What is the windows forwarder looking for? I set the indexer not to verify client certs but does the forwarder need a client certificate (self-signed or otherwise) generated regardless to use SSL ?

Labels (1)
Tags (1)
0 Karma
1 Solution

shocko
Contributor

So I resolved my specific issue as follows:

Since my indexer is using a PKI signed certificate and that PKI has a Root CA and Issuing CA I had to add the Issuing CA public cert and Root CA to a .PEM file (in that order) and drop onto my forwarder

In outputs.conf I then reference it as follows:

[tcpout:test-ssl-1]
disabled = 0
server = indexer1.mydomain.com:9998
useSSL = true
useClientSSLCompression = true

sslVerifyServerCert = false
sslRootCAPath = C:\Program Files\SplunkUniversalForwarder\etc\auth\CA_Chain.pem

So I have a working setup with the indexer using a PKI signed certificate and the forwarder without defining any client certs. Even though sslVerifyServerCert is set to false I still need to supply sslRootPath. Again, I don't know why as it doesn't make sense to me 😐

My takeaways:

  1. In order for the forwarder to ship events to the indexer over SSL a client certificate does not need be defined on the forwarder outputs.conf files
  2. The statement regarding the password for the client PEM file not being encrypted if it's defined n inputs.conf or outputs.conf outside of /etc/system/local/ does not appear to be true in 8.2.5 as my passwords are getting encrypted in those config files under the apps directory when the forwarder restarts
  3. If you wish to verify the indexer cert and it is using a PKI then you must point the forwarder at a PEM file that contains all CAs in that chain from bottom to top

View solution in original post

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Your forwarder would need SSL certs and configurations as well to enable SSL communication with your SSL enabled indexer. This documentation will give you all the details: https://docs.splunk.com/Documentation/Splunk/8.2.5/Security/ConfigureSplunkforwardingtousesignedcert...

0 Karma

shocko
Contributor

Since I have told the indexer to ignore client certs what does the client need them for?

0 Karma

Stefanie
Builder

@shocko 

I had similar problems with my set up for SSL. 

Are you able to run the command:

>openssl.exe rsa -in "C:\Program Files\Splunk\etc\auth\mycert\my.pem" -text

 

Try following the steps listed here if you haven't

https://docs.splunk.com/Documentation/Splunk/8.2.5/Security/Troubleshootyouforwardertoindexerauthent...

0 Karma

shocko
Contributor

So I have verified the indexer is listening on TCP 9998 and has my PKI certificate bound to it. I don't understand why the forwarder needs any client certificates to use SSL. I just want to ensure the data is forwarded over SSL. Why would I need a client certificate for this? I'm using deployment server to deploy my apps so I also don't want to specify the password for the client certificate private key in the outputs.conf. 

0 Karma

Stefanie
Builder

@shocko wrote:

So I have verified the indexer is listening on TCP 9998 and has my PKI certificate bound to it. I don't understand why the forwarder needs any client certificates to use SSL. I just want to ensure the data is forwarded over SSL. Why would I need a client certificate for this? 


For the SSL connection to the indexers the forwarder requires a certificate. The clientCert is used to "turn on" SSL connections. That's my assumption. 

You can use the certificate you created for your indexers to use on your forwarders.

 


@shocko wrote:

I'm using deployment server to deploy my apps so I also don't want to specify the password for the client certificate private key in the outputs.conf. 


Splunk doesn't support setting up SSL certificates in apps for this very reason anymore. It took me a long time of trial and error before someone @ Splunk told me this. You'll need to place your certificate somewhere in $SPLUNK_HOME/etc/auth/(folder) and your outputs.conf in $SPLUNK_HOME/etc/system/local

 

 

 

Touching back on your error you received on the splunkd.log on your forwarder, if you restart the indexer do you see where your indexer is successfully accepting SSL? It might say something like 

port 9998 is reserved for splunk 2 splunk (SSL)

 

 

 

shocko
Contributor

So I resolved my specific issue as follows:

Since my indexer is using a PKI signed certificate and that PKI has a Root CA and Issuing CA I had to add the Issuing CA public cert and Root CA to a .PEM file (in that order) and drop onto my forwarder

In outputs.conf I then reference it as follows:

[tcpout:test-ssl-1]
disabled = 0
server = indexer1.mydomain.com:9998
useSSL = true
useClientSSLCompression = true

sslVerifyServerCert = false
sslRootCAPath = C:\Program Files\SplunkUniversalForwarder\etc\auth\CA_Chain.pem

So I have a working setup with the indexer using a PKI signed certificate and the forwarder without defining any client certs. Even though sslVerifyServerCert is set to false I still need to supply sslRootPath. Again, I don't know why as it doesn't make sense to me 😐

My takeaways:

  1. In order for the forwarder to ship events to the indexer over SSL a client certificate does not need be defined on the forwarder outputs.conf files
  2. The statement regarding the password for the client PEM file not being encrypted if it's defined n inputs.conf or outputs.conf outside of /etc/system/local/ does not appear to be true in 8.2.5 as my passwords are getting encrypted in those config files under the apps directory when the forwarder restarts
  3. If you wish to verify the indexer cert and it is using a PKI then you must point the forwarder at a PEM file that contains all CAs in that chain from bottom to top
0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...