Forwarder to Indexer secure communication using self signed certs?

newportknight — Tue, 28 Mar 2023 14:11:13 GMT

Hi,

I am trying to get secure comms between a Forwarder and Indexer up and running using self signed certs but depite following the relevant guides (https://docs.splunk.com/Documentation/Splunk/8.2.1/Security/Howtoself-signcertificates) I keep ending up with the same problem.

I'm generating the self signed cert on a deployment server, creating the RootCA cert, servercert and serverprivate key before transferring them to the Indexer and Forwarder. Once on these I'm creating a newserver cert by combining the 3 files.

I've also created the relevant inputs.conf, outputs.conf and server.conf files using the config guide. It does say to use "password = <string>" in both inputs and outputs conf files but this kicks up an error as it is deprecated so I've used "sslPassword" instead.

After restarting splunkd in the splunkd log on the Indexer I'm getting:

ERROR TcpInputProc - Error encountered for connection from src=10.1.1.34:50772. error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol

I've tried searching for the error and trying various other fixes e.g specifying sslVersions or cipherSuite but I'm still getting the above error.

Could any one offer some help as to where I may be going wrong please?

I've copied the conf files and some outputs from the splund.logs.

Forwarder outputs.conf

[tcpout:group1]

server = 10.1.1.20:9997

disabled = 0

clientCert = /opt/splunk/etc/auth/mycerts/myNewServerCertificate.pem
sslPassword = <key used to generate myServerPrivateKey.key>

useClientSSLCompression = true

Forwarder server.conf

[sslConfig]
sslRootCAPath = /opt/splunk/etc/auth/mycerts/myCACertificate.pem

Forwarder splunkd log

cat /opt/splunk/var/log/splunk/splunkd.log | grep SSL
07-08-2021 10:19:22.919 +0100 INFO loader - Setting SSL configuration.
07-08-2021 10:19:22.919 +0100 INFO loader - Server supporting SSL versions SSL3,TLS1.0,TLS1.1,TLS1.2
07-08-2021 10:19:46.393 +0100 INFO MongodRunner - Using mongod command line --sslMode requireSSL
07-08-2021 10:19:47.957 +0100 INFO TcpInputProc - Creating fwd data Acceptor for IPv4 port 9997 with Non-SSL

cat /opt/splunk/var/log/splunk/splunkd.log | grep TcpOut
07-08-2021 10:43:42.172 +0100 INFO TcpOutputProc - Found currently active indexer. Connected to idx=10.1.1.20:9997, reuse=1.

Indexer inputs.conf

[splunktcp-ssl:9997]
disabled = 0

[SSL]
serverCert = /opt/splunk/etc/auth/mycerts/myNewServerCertificate.pem

sslPassword = <key used to generate myServerPrivateKey.key>

requireClientCert = false

useSSLCompression = false

Indexer server.conf

[sslConfig]
sslPassword = $7$YNwWFOGvWECUWkppnTLseT5sGq3wJs72wGEjlZuHDphTK3Jty2nhPQ==

sslRootCAPath = /opt/splunk/etc/auth/mycerts/myCACertificate.pem

Indexer splunkd.log

cat /opt/splunk/var/log/splunk/splunkd.log | grep SSL
07-08-2021 10:29:02.382 +0100 INFO ServerConfig - SSL session cache path enabled 0 session timeout on SSL server 300.000
07-08-2021 10:29:02.520 +0100 INFO loader - Setting SSL configuration.
07-08-2021 10:29:02.520 +0100 INFO loader - Server supporting SSL versions SSL3,TLS1.0,TLS1.1,TLS1.2
07-08-2021 10:29:03.093 +0100 INFO MongodRunner - Using mongod command line --sslMode requireSSL
07-08-2021 10:29:04.886 +0100 INFO TcpInputConfig - Creating FwdDataSSLConfig SSL context. Will open port=IPv4 port 9997 with compression=1
07-08-2021 10:29:04.914 +0100 INFO TcpInputConfig - IPv4 port 9997 is reserved for splunk 2 splunk (SSL)
07-08-2021 10:29:04.915 +0100 INFO TcpInputProc - Creating fwd data Acceptor for IPv4 port 9997 with SSL
07-08-2021 10:32:14.117 +0100 ERROR TcpInputProc - Error encountered for connection from src=10.1.1.34:50770. error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
07-08-2021 10:32:14.118 +0100 ERROR TcpInputProc - Error encountered for connection from src=10.1.1.34:50772. error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol

cat /opt/splunk/var/log/splunk/splunkd.log | grep Tcp
07-08-2021 10:29:04.885 +0100 INFO TcpInputConfig - IPv4 port 9997 is reserved for splunk 2 splunk
07-08-2021 10:29:04.886 +0100 INFO TcpInputConfig - Creating FwdDataSSLConfig SSL context. Will open port=IPv4 port 9997 with compression=1
07-08-2021 10:29:04.914 +0100 INFO TcpInputConfig - IPv4 port 9997 is reserved for splunk 2 splunk (SSL)
07-08-2021 10:29:04.915 +0100 INFO TcpInputProc - Creating fwd data Acceptor for IPv4 port 9997 with SSL
07-08-2021 10:29:05.308 +0100 INFO TcpOutputProc - _isHttpOutConfigured=NOT_CONFIGURED
07-08-2021 10:32:14.117 +0100 ERROR TcpInputProc - Error encountered for connection from src=10.1.1.34:50770. error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
07-08-2021 10:32:14.118 +0100 ERROR TcpInputProc - Error encountered for connection from src=10.1.1.34:50772. error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol

Re: Forwarder to Indexer secure communication using self signed certs

triptraptresko — Tue, 28 Mar 2023 13:02:40 GMT

TLDR: I was missing
1. outputs.conf on the sender (client)
useSSL = true

Thank you to michael_bates_1 , in thread https://community.splunk.com/t5/Getting-Data-In/Why-am-I-having-trouble-with-TLS/m-p/634513/highlight/false#M108573

After following the documentation on how to enable ssl between forwarders and indexers, i got the error

ERROR TcpInputProc - Error encountered for connection from src=10.1.1.34:50772. error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol

In the documentation: https://docs.splunk.com/Documentation/Splunk/9.0.1/Security/ConfigureSplunkforwardingtousesignedcertificates#Configuration_file_examples_for_configuring_TLS_certificates_on_receiving_indexers
, it specifies

[SSL]
requireClientCert = true

Which if you drop, will affect your outputs.conf -> useSSL.
It says if requireClientCert is defined, then useSSL will be true.
In my case, I mindlessly thought you could set requrieClientCert=false...

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Outputsconf
#----Secure Sockets Layer (SSL) Settings----

# To set up SSL on the forwarder, set the following setting/value pairs.
# If you want to use SSL for authentication, add a stanza for each receiver
# that must be certified.

useSSL = <true|false|legacy>
* Whether or not the forwarder uses SSL to connect to the receiver, or relies
  on the 'clientCert' setting to be active for SSL connections.
* You do not need to set 'clientCert' if 'requireClientCert' is set to
  "false" on the receiver.
* A value of "true" means the forwarder uses SSL to connect to the receiver.
* A value of "false" means the forwarder does not use SSL to connect to the
  receiver.
* The special value "legacy" means the forwarder uses the 'clientCert' property to
  determine whether or not to use SSL to connect.
* Default: legacy

topic Forwarder to Indexer secure communication using self signed certs? in Getting Data In

Forwarder to Indexer secure communication using self signed certs?

Re: Forwarder to Indexer secure communication using self signed certs