Could I get some advice on configuring self-signed certs between Forwarder and Indexer?

[liquidclay23]

Hello Everyone,

I am having trouble configuring self-signed certs and was wondering if I could possibly get some advice.

I am doing this in a test environment with the express purpose of replicating the configurations listed in the Splunk docs (.../Splunk/7.1.3/Security/Howtoself-signcertificates)

These configs are being performed on a deployment server. The deployment server's splunk.secret was replicabed to all boxes upon initial install. This splunk 7.1.2 on RHEL 7.

Currently I am getting the following error:
- ERROR TcpInputProc - Error encountered for connection from src=10.0.0.1:36014. error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
- WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read server certificate B', alert_description='unknown CA'.

Here is my configuration:

Create a key to sign your certificates.

/opt/splunk/bin/splunk cmd openssl genrsa -aes256 -out myCAPrivateKey.key 2048
splunk_$certs

• Generate a new Certificate Signing Request (CSR) When prompted, create a password for the key.
  /opt/splunk/bin/splunk cmd openssl req -new -key myCAPrivateKey.key -out myCACertificate.csr
  splunk_$certs

• Anything not specified is left default/blank
  Country Name (2 letter code) [AU]:US
  State or Province Name (full name) [Some-State]:DC
  Locality Name (eg, city) []:Washington
  Organization Name (eg, company) [Internet Widgits Pty Ltd]:MyTestOrg
  Organizational Unit Name (eg, section) []:SecDiv
  A challenge password []:splunk_$certs
  Common Name (e.g. server FQDN or YOUR name) []:Deployment_Server

• Use the CSR myCACertificate.csr to generate the public certificate:
  /opt/splunk/bin/splunk cmd openssl x509 -req -in myCACertificate.csr -sha512 -signkey myCAPrivateKey.key -CAcreateserial -out myCACertificate.pem -days 10950
  splunk_$certs

Create the server certificate for the search head to forward its data to the indexers

/opt/splunk/bin/splunk cmd openssl genrsa -aes256 -out myServerPrivateKey.key 2048
splunk_$certs

• Generate and sign a new server certificate

/opt/splunk/bin/splunk cmd openssl req -new -key myServerPrivateKey.key -out myServerCertificate.csr
splunk_$certs

• Anything not specified is left default/blank Country Name (2 letter code) [AU]:US State or Province Name (full name) [Some-State]:DC Locality Name (eg, city) []:Washington Organization Name (eg, company) [Internet Widgits Pty Ltd]:MyTestOrg Organizational Unit Name (eg, section) []:SecDiv Common Name (e.g. server FQDN or YOUR name) []:Search_Head A challenge password []:splunk_$certs

/opt/splunk/bin/splunk cmd openssl x509 -req -in myServerCertificate.csr -SHA256 -CA myCACertificate.pem -CAkey myCAPrivateKey.key -CAcreateserial -out myServerCertificate.pem -days 1095
splunk_$certs

• Create a single PEM file
• Once you have your certificates, you must combine the server certificate and your keys into a single file that Splunk software can use.

cat myServerCertificate.pem myServerPrivateKey.key myCACertificate.pem > myNewServerCertificate.pem

• The CA cert is copied to a deployment app so it can be reused. The Search_Head/server certs are moved.
• The 'dev_transit_forwarder_certs' app is transfered to the search head via the deployment server

cp myCA* /opt/splunk/etc/deployment-apps/dev_transit_forwarder_certs/splunk
mv myNew* /opt/splunk/etc/deployment-apps/dev_transit_forwarder_certs/splunk
mv myServer* /opt/splunk/etc/deployment-apps/dev_transit_forwarder_certs/splunk

Create the server certificate for the search head to forward its data to the indexers

/opt/splunk/bin/splunk cmd openssl genrsa -aes256 -out myServerPrivateKey.key 2048
splunk_$certs

• Generate and sign a new server certificate

/opt/splunk/bin/splunk cmd openssl req -new -key myServerPrivateKey.key -out myServerCertificate.csr
splunk_$certs

• Anything not specified is left default/blank Country Name (2 letter code) [AU]:US State or Province Name (full name) [Some-State]:DC Locality Name (eg, city) []:Washington Organization Name (eg, company) [Internet Widgits Pty Ltd]:MyTestOrg Organizational Unit Name (eg, section) []:SecDiv Common Name (e.g. server FQDN or YOUR name) []:Indexer A challenge password []:splunk_$certs

/opt/splunk/bin/splunk cmd openssl x509 -req -in myServerCertificate.csr -SHA256 -CA myCACertificate.pem -CAkey myCAPrivateKey.key -CAcreateserial -out myServerCertificate.pem -days 1095
splunk_$certs

• Create a single PEM file
• Once you have your certificates, you must combine the server certificate and your keys into a single file that Splunk software can use.

cat myServerCertificate.pem myServerPrivateKey.key myCACertificate.pem > myNewServerCertificate.pem

• The certs are copied to a deployment app.
• The 'dev_transit_indexer_certs' app is transfered to two indexers via the deployment server

cp myCA* /opt/splunk/etc/deployment-apps/dev_transit_indexer_certs/splunk
mv myNew* /opt/splunk/etc/deployment-apps/dev_transit_indexer_certs/splunk
mv myServer* /opt/splunk/etc/deployment-apps/dev_transit_indexer_certs/splunk

Search Head configurations

/dev_hf_outputs/local/server.conf

[sslConfig]
sslRootCAPath = /opt/splunk/etc/apps/dev_transit_forwarder_certs/splunk/myCACertificate.pem

/dev_hf_outputs/local/outputs.conf

[indexAndForward]
index = false

[tcpout]
defaultGroup = dev_indexers
indexAndForward = false

[tcpout:dev_indexers]
server = 10.0.0.10:9996,10.0.0.11:9996
disabled = 0

[tcpout:splunkssl]
clientCert = /opt/splunk/etc/apps/dev_transit_forwarder_certs/splunk/myNewServerCertificate.pem
sslPassword = splunk_$certs
sslVerifyServerCert = false

Indexer configurations

/dev_indexers_inputs/local/server.conf

[sslConfig]
sslRootCAPath = /opt/splunk/etc/apps/dev_transit_indexer_certs/splunk/myCACertificate.pem

/dev_indexers_inputs/local/inputs.conf

[splunktcp-ssl:9996]
disabled = 0

[SSL]
serverCert = /opt/splunk/etc/apps/dev_transit_indexer_certs/splunk/myNewServerCertificate.pem
sslPassword = splunk_$certs
requireClientCert= false

[liquidclay23]

I got this mostly working by correcting my outputs.conf. Incase it helps anyone

[tcpout]
defaultGroup = dev_indexers
indexAndForward = false
useACK = true

[tcpout:dev_indexers]
server = 10.0.0.10:9996,10.0.0.11:9996
disabled = 0
sslPassword = splunk_$certs
sslVerifyServerCert = true
useClientSSLCompression = true
sslCertPath = $SPLUNK_HOME/etc/apps/dev_transit_forwarder_certs/splunk/myNewServerCertificate.pem

[inventsekar]

on SSL areas, i got many questions as well.. i wrote this comment sometime back..
"SSL is one of the most difficult areas and its the least documented on splunk docs"