Can you help me with an issue I'm having with setting up a forwarder to Indexer SSL Comms

[damode]

Initially I had started to set up Forward-to-indexer SSL setup using self signed certificate. However, I was getting the below error on the indexer in the Splunkd log,

ERROR TcpInputProc - Error encountered for connection from src=192.168.14.10:49497. error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number

Thinking that I may have followed the process incorrectly, I resorted to setup the SSL using Splunk Default certs. however, I am still getting the same error.

I have tried changing the SSLversions to "* ,-ssl2" but retracted and kept the default which is tls1.2

I am using Splunk version 7.1.2 on both Indexer and UF.

Please advise.

[harsmarvania57]

Hi @damode,

After implementing SSL certificates on Indexer and Forwarder, have you tried to connect from forwarder to indexer using command openssl s_client -connect {server}:{port} ? Can you please provide output of this command (Please mask any sensitive data) ?

[damode]

Hi Harsh,
Here are the details,

C:\Windows\system32>splunk cmd openssl s_client -connect 192.168.14.1:9997
CONNECTED(000000F0)
depth=1 C = US, ST = CA, L = San Francisco, O = Splunk, CN = SplunkCommonCA, emailAddress = support@splunk.com
verify error:num=19:self signed certificate in certificate chain
---
Certificate chain
 0 s:/CN=SplunkServerDefaultCert/O=SplunkUser
   i:/C=US/ST=CA/L=San Francisco/O=Splunk/CN=SplunkCommonCA/emailAddress=support@splunk.com
 1 s:/C=US/ST=CA/L=San Francisco/O=Splunk/CN=SplunkCommonCA/emailAddress=support@splunk.com
   i:/C=US/ST=CA/L=San Francisco/O=Splunk/CN=SplunkCommonCA/emailAddress=support@splunk.com
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDMjCCAhoCCQDQdsr0LOW0rTANBgkqhkiG9w0BAQsFADB/MQswCQYDVQQGEwJV
UzELMAkGA1UECAwCQ0ExFjAUBgNVBAcMDVNhbiBGcmFuY2lzY28xDzANBgNVBAoM
BlNwbHVuazEXMBUGA1UEAwwOU3BsdW5rQ29tbW9uQ0ExITAfBgkqhkiG9w0BCQEW
EnN1cHBvcnRAc3BsdW5rLmNvbTAeFw0xODA3MTgwMTU2NDhaFw0yMTA3MTcwMTU2
NDhaMDcxIDAeBgNVBAMMF1NwbHVua1NlcnZlckRlZmF1bHRDZXJ0MRMwEQYDVQQK
DApTcGx1bmtVc2VyMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA2pd4
Dm5hIWqTwum95KLpNLyW6Bh0vLDWv8N11IZvvmBalr9Mxm4zrdgzeIv8mc26dyg0
Q4D9HyNKGzS1VYDHACohqFpwy6tIXA090TF4jX44vAAL1aAuamKKeHXfudectPTn
PEhwr8hUsNox4WcN9zI9vlBnDFXuByCaA3dMXOsihB/1kFLna4PCBD/VgM8bsi9Q
QWrFhuHWFuNmjTTCKGKeAB8mcoOkDNiSK5pKknMqh+SAfljgioYEdqGLhBUD+cTf
d5feJCWh3G0Hy1+xuC0r8mskcrWGMF0NXLRziskxV4Y3MREfhbRESYpu1i7jTC2X
Zd5VLaJuupGPsgyDFQIDAQABMA0GCSqGSIb3DQEBCwUAA4IBAQCJEj9Zm0E7MEAD
7c1klOS1ZqtPEEUNULiISohJQcTOG17cbl1BZlmLGCgHgmAXhuWC3xoYzwMa6Asm
vrsBtccyDvtXoudssjmzcdR7bd13ply3voMCXtDMTyZcgGzQSTuQtyxnVmzeYfeu
/+ZTP5bck87uKy4v5ElMnwXGx6hATfuaKDcJHPONwU++HBxfMW/mSNhLPQEV7kzT
dcsgB9EFnIYxD7LAVaIAmjTaP5Us5IQ+x5Hw+3nAYW2EvY5WtUEnwRB5BmMPE6eQ
Wh9RrdBxA8GVoodiFnjkEX/BqAo8akE3mECBcgEbg2zP20KWODTlq3ubfkdVbVSn
IpdY5e63
-----END CERTIFICATE-----
subject=/CN=SplunkServerDefaultCert/O=SplunkUser
issuer=/C=US/ST=CA/L=San Francisco/O=Splunk/CN=SplunkCommonCA/emailAddress=support@splunk.com
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2387 bytes and written 441 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: zlib compression
Expansion: zlib compression
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 219FFC9E3E943B40B0362FA4EDBCC763873F6FE0B1055F0AE35850A0B2378BE0
    Session-ID-ctx:
    Master-Key: 76CA9533CD8C2C755EBD9C81EAF9A3967CEA54574FF557C63F0C68017612EA655FC58F56831E58F29C51575DB84C978C
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - f7 7d 00 e8 4e 4c 25 f9-a1 d8 fc 89 2b a7 e5 09   .}..NL%.....+...
    0010 - 84 bf 4f ec 38 0a b1 d9-84 cf 15 04 16 ca f8 d5   ..O.8...........
    0020 - 8d bc cf 45 fb 2f 26 9c-2f 23 ff 69 ef 02 33 77   ...E./&./#.i..3w
    0030 - 52 56 b8 d0 98 6d c2 19-79 31 b0 5a 7c 80 56 8d   RV...m..y1.Z|.V.
    0040 - 42 1a be a0 2f a7 ef 83-8f 03 d2 75 be 8e a8 f0   B.../......u....
    0050 - 6d 5a d7 b1 db 6c 66 de-6b 5f 7d 49 0a 0e 5b 73   mZ...lf.k_}I..[s
    0060 - f9 30 95 5c 55 c7 52 83-65 35 d5 fc 86 19 01 69   .0.\U.R.e5.....i
    0070 - cb 8f c5 7c cc c5 3a 6d-f7 78 98 34 04 b5 66 58   ...|..:m.x.4..fX
    0080 - 09 20 3b 97 45 0c 0d e3-ba 04 50 a8 57 b9 ef 6e   . ;.E.....P.W..n
    0090 - ac 6f 08 9b ae 85 f9 a0-68 7d 36 26 74 90 5d 1f   .o......h}6&t.].

    Compression: 1 (zlib compression)
    Start Time: 1536899254
    Timeout   : 300 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
---
read:errno=0

On the forwarder Splunkd log, I get this message,

DEBUG TcpOutputProc - channel not registered yet
DEBUG TcpOutputProc - Connection not available. Waiting for connection

and on indexer,
INFO TcpInputProc - Creating fwd data Acceptor for IPv4 port 9997 with SSL

[harsmarvania57]

Connection looks good, have you configured Indexer and Forwarder as below with your self signed certificate (Both self signed certificate should be signed with same CA) ?

Indexer server.conf

[sslConfig]
sslPassword = <default_encrypyed_password>
sslRootCAPath = <PATH_OF_ROOTCA_CERTIFICATE>

Indexer inputs.conf

[splunktcp-ssl://9997]

[SSL]
serverCert = <PATH_OF_INDEXER_CERTIFICATE>
sslPassword = <INDEXER_CERTIFICATE_KEY_FILE_PASSWORD>
requireClientCert = true

Forwarder server.conf

[sslConfig]
sslPassword = <default_encrypyed_password>
sslRootCAPath = <PATH_OF_ROOTCA_CERTIFICATE>

Forwarder outputs.conf

[tcpout]
defaultGroup = indexers
useACK = true

[tcpout:indexers]
server = <INDEXER>:<PORT>
sslPassword = <FORWARDER_CERTIFICATE_KEY_FILE_PASSWORD>
sslCertPath = <PATH_OF_FORWARDER_CERTIFICATE>
sslVerifyServerCert = true
useClientSSLCompression = true

Once you implement this, restart splunk on Indexer and Forwarder and then try to connect with below command from forwarder to indexer, it should connect with without error.

/opt/splunk/bin/splunk cmd openssl s_client -connect <INDEXER>:<SSLPORT> -cert <PATH_OF_FORWARDER_CERTIFICATE> -CAfile <PATH_OF_ROOTCA_CERTIFICATE>

Detailed documentation by Splunk provided here http://docs.splunk.com/Documentation/Splunk/7.1.3/Security/ConfigureSplunkforwardingtousesignedcerti...

[damode]

Hi Harsh,

I have configured with above with Splunk default CA.
I was able to establish a successful comms from the forwarder to Indexer using

/opt/splunk/bin/splunk cmd openssl s_client -connect <INDEXER>:<SSLPORT> -cert <PATH_OF_FORWARDER_CERTIFICATE> -CAfile <PATH_OF_ROOTCA_CERTIFICATE>

However, I am still getting the exact same error.

[damode]

When I run the below command,

C:\Windows\system32>splunk cmd openssl
s_client -connect 192.168.14.30:9997
-tls1_2
the connection is successful but with the below errors,

CONNECTED(000000F0)
    depth=1 C = US, ST = CA, L = San Francisco, O = Splunk, CN = SplunkCommonCA, emailAddress = support@splunk.com
    verify error:num=19:self signed certificate in certificate chain
    164:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:.\ssl\s3_pkt.c:1498:SSL alert number 40
    164:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:.\ssl\s3_pkt.c:659: ---

However, when I run the command provided by you, it also connects but with no error.

I also tried to change the ssl version to -

sslVersions = *,-ssl2

However, after doing that, I get this error,
ERROR TcpInputProc - Error encountered for connection from src=192.168.14.10:50890. error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol

[harsmarvania57]

Can you please provide your inputs.conf from Indexer and outputs.conf from Forwarder, additionally it is not recommended to use Splunk default certificate instead you need to use Internal Signed Certificate or 3rd party certificate.

[damode]

Forwarder - outputs.conf

[tcpout:group1]
server = 192.168.14.30:9997
clientCert = C:\Program Files\SplunkUniversalForwarder\etc\auth\server.pem
sslPassword = password
sslVerifyServerCert = true
useClientSSLCompression = true
sslVersions = *,-ssl2

server.conf

[sslConfig]
sslPassword = password
sslRootCAPath = C:\Program Files\SplunkUniversalForwarder\etc\auth\cacert.pem
sslVersions = *,-ssl2

Indexer
inputs.conf

[splunktcp-ssl:9997]
disabled = 0

[SSL]
serverCert = C:\Program Files\Splunk\etc\auth\server.pem
sslVersions = *,-ssl2
sslPassword = password
requireClientCert = true

server.conf

[sslConfig]
sslPassword = password
sslRootCAPath = C:\Program Files\Splunk\etc\auth\cacert.pem
sslVersions = *,-ssl2

I had initially started with self-signed certs but to make it less complicated, I am using Splunk default certs. Atleast once it starts working properly, I will move to self signed.

[harsmarvania57]

You need to try with your own internal certificate instead of Splunk Default certificate because on Indexer and Forwarder there will be different cacert.pem, so when forwarder send encrypted request to Indexer, cacert on Indexer will not able to decrypt it because Indexer has different cacert. So as I mentioned earlier both server.pem (On Indexer and Forwarder) should be signed with same CA certificate and that same CA certificate must be present on both the servers (Indexer and Forwarder)

[damode]

Before using the Splunk Default Certs, I had tried setting up SSL using self signed itself, as I have mentioned in my post. However, because that was giving me the same errors, I thought to make it simpler by using Splunk defaults.

[damode]

Based on this link - link text

It seems to be an issue with the openssl version.