Splunk - Forwarder 2 Indexer - Self Signed Certificate Failure

[jefffredericks]

I've set up a dev 9.2 Splunk environment. And I'm trying to use a self-signed cert to secure forwarding. But every time I attempt to connect the UF to the Indexing server it fails -_-

I've tried a lot of permutations of the below. All ultimately ending with the forwarder unable to connect to the indexing server. I've made sure permissions are set to 6000 for cert and key. Made sure the Forwarder and Indexer have seperate common names. And created multiple cert types. But I'm at a bit of a loss as to what I need to do to get the forwarder and indexer to connect over a self signed certificate.

Any help is incredibly appreciated.

Below is some of what I've attempted. Trying to not make this post multiple pages long X)

1. Simple TLS Configuration

• Generating Indexer Certs:

  openssl genrsa -out indexer.key 2048
  
  openssl req -new -x509 -key indexer.key -out indexer.pem -days 1095 -sha256
  
  cat indexer.pem indexer.key > indexer_combined.pem
  
  Note: I keep reading that the cert and key need to be 1 file.  But I"m not sure on this.

• Generating Forwarder Certs:

  openssl genrsa -out forwarder.key 2048
  
  openssl req -new -x509 -key forwarder.key -out forwarder.pem -days 1095 -sha256
  
  cat forwarder.pem forwarder.key > forwarder_combined.pem

• Indexer Configuration:

  [SSL]
  serverCert = /opt/tls/indexer_combined.pem
  sslPassword = random_string
  requireClientCert = false
  
  [splunktcp-ssl:9997]
  compressed = true

  Outcome: Indexer listens on port 9997 for encrypted communications.

• Forwarder Configuration

  [tcpout]
  defaultGroup = splunkssl
  
  [tcpout:splunkssl]
  server = 192.168.110.178:9997
  compressed = true
  
  [tcpout-server://192.168.110.178:9997]
  sslCertPath =/opt/tls/forwarder_combined.pem
  sslPassword = random_string
  sslVerifyServerCert = false

  Outcome: Forwarder fails to communicate with Indexer

Logs from Forwarder:

ERROR TcpInputProc [27440 FwdDataReceiverThread] - Error encountered for connection from src=192.168.110.26:33522. error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol

Testing with openssl s_client:

Command: openssl s_client -connect 192.168.110.178:9997 -cert forwarder_combined.pem -key forwarder.key

Output: Unknown CA ( I didn't write the exact message in my notes, but it generally says the CA is unknown.)

Note: Not sure if I need to add sslVersions = tls1.2, but that seems outside of the scope of the issue.

Troubleshooting connect, running openssl s_client raw:

Command: openssl s_client -connect 192.168.110.178:9997

Output received:

CONNECTED(00000003)
Can't use SSL_get_servername

Full s_client message is here: https://pastebin.com/z9gt7bhz

1. Further Troubleshooting

• Added Indexers self-signed certificate to forwarder

  ...
  sslPassword = random_string
  sslVerifyServerCert = true
  sslRootCAPath = /opt/tls/indexer_combined.pem

  Outcome: same error message.

Testing with s_client:

Command: openssl s_client -connect 192.168.110.178:9997 -CAfile indexer_combined.pem

Connecting to 192.168.110.178 CONNECTED(00000003) Can't use SSL_get_servername

Full s_client message is here: https://pastebin.com/BcDvJ2Fs

[PickleRick]

OK. A few things here and there.

1. The format for the cert file (for inputs and generally for all splunkd-related activity except the webui (which can be a bit confusing sometimes) is:

<subject cert (i.e. your forwarder or splunk server>
<private key>
<CA chain (if needed)>

(all of them PEM-encoded)

2. If you don't want to authenticate forwarder with a cert there's no point of generating one for it.

3. The "SSL23_GET_CLIENT_HELLO:unknown protocol" message is a fairly generic one. Check the indexer's logs for anything regarding connection from the forwarder's IP. This should tell you more.