Security
Highlighted

Could I get some advice on configuring self-signed certs between Forwarder and Indexer?

Engager

Hello Everyone,

I am having trouble configuring self-signed certs and was wondering if I could possibly get some advice.

I am doing this in a test environment with the express purpose of replicating the configurations listed in the Splunk docs (.../Splunk/7.1.3/Security/Howtoself-signcertificates)

These configs are being performed on a deployment server. The deployment server's splunk.secret was replicabed to all boxes upon initial install. This splunk 7.1.2 on RHEL 7.

Currently I am getting the following error:
- ERROR TcpInputProc - Error encountered for connection from src=10.0.0.1:36014. error:1408F10B:SSL routines:SSL3GETRECORD:wrong version number
- WARN SSLCommon - Received fatal SSL3 alert. sslstate='SSLv3 read server certificate B', alertdescription='unknown CA'.

Here is my configuration:

Create a key to sign your certificates.

/opt/splunk/bin/splunk cmd openssl genrsa -aes256 -out myCAPrivateKey.key 2048
splunk_$certs

  • Generate a new Certificate Signing Request (CSR) When prompted, create a password for the key.
    /opt/splunk/bin/splunk cmd openssl req -new -key myCAPrivateKey.key -out myCACertificate.csr
    splunk_$certs

  • Anything not specified is left default/blank
    Country Name (2 letter code) [AU]:US
    State or Province Name (full name) [Some-State]:DC
    Locality Name (eg, city) []:Washington
    Organization Name (eg, company) [Internet Widgits Pty Ltd]:MyTestOrg
    Organizational Unit Name (eg, section) []:SecDiv
    A challenge password []:splunk$certs
    Common Name (e.g. server FQDN or YOUR name) []:Deployment
    Server

  • Use the CSR myCACertificate.csr to generate the public certificate:
    /opt/splunk/bin/splunk cmd openssl x509 -req -in myCACertificate.csr -sha512 -signkey myCAPrivateKey.key -CAcreateserial -out myCACertificate.pem -days 10950
    splunk_$certs

Create the server certificate for the search head to forward its data to the indexers

/opt/splunk/bin/splunk cmd openssl genrsa -aes256 -out myServerPrivateKey.key 2048
splunk_$certs

  • Generate and sign a new server certificate

/opt/splunk/bin/splunk cmd openssl req -new -key myServerPrivateKey.key -out myServerCertificate.csr
splunk_$certs

  • Anything not specified is left default/blank Country Name (2 letter code) [AU]:US State or Province Name (full name) [Some-State]:DC Locality Name (eg, city) []:Washington Organization Name (eg, company) [Internet Widgits Pty Ltd]:MyTestOrg Organizational Unit Name (eg, section) []:SecDiv Common Name (e.g. server FQDN or YOUR name) []:SearchHead A challenge password []:splunk$certs

/opt/splunk/bin/splunk cmd openssl x509 -req -in myServerCertificate.csr -SHA256 -CA myCACertificate.pem -CAkey myCAPrivateKey.key -CAcreateserial -out myServerCertificate.pem -days 1095
splunk_$certs

  • Create a single PEM file
  • Once you have your certificates, you must combine the server certificate and your keys into a single file that Splunk software can use.

cat myServerCertificate.pem myServerPrivateKey.key myCACertificate.pem > myNewServerCertificate.pem

  • The CA cert is copied to a deployment app so it can be reused. The Search_Head/server certs are moved.
  • The 'devtransitforwarder_certs' app is transfered to the search head via the deployment server

cp myCA* /opt/splunk/etc/deployment-apps/devtransitforwardercerts/splunk
mv myNew* /opt/splunk/etc/deployment-apps/dev
transitforwardercerts/splunk
mv myServer* /opt/splunk/etc/deployment-apps/devtransitforwarder_certs/splunk

Create the server certificate for the search head to forward its data to the indexers

/opt/splunk/bin/splunk cmd openssl genrsa -aes256 -out myServerPrivateKey.key 2048
splunk_$certs

  • Generate and sign a new server certificate

/opt/splunk/bin/splunk cmd openssl req -new -key myServerPrivateKey.key -out myServerCertificate.csr
splunk_$certs

  • Anything not specified is left default/blank Country Name (2 letter code) [AU]:US State or Province Name (full name) [Some-State]:DC Locality Name (eg, city) []:Washington Organization Name (eg, company) [Internet Widgits Pty Ltd]:MyTestOrg Organizational Unit Name (eg, section) []:SecDiv Common Name (e.g. server FQDN or YOUR name) []:Indexer A challenge password []:splunk_$certs

/opt/splunk/bin/splunk cmd openssl x509 -req -in myServerCertificate.csr -SHA256 -CA myCACertificate.pem -CAkey myCAPrivateKey.key -CAcreateserial -out myServerCertificate.pem -days 1095
splunk_$certs

  • Create a single PEM file
  • Once you have your certificates, you must combine the server certificate and your keys into a single file that Splunk software can use.

cat myServerCertificate.pem myServerPrivateKey.key myCACertificate.pem > myNewServerCertificate.pem

  • The certs are copied to a deployment app.
  • The 'devtransitindexer_certs' app is transfered to two indexers via the deployment server

cp myCA* /opt/splunk/etc/deployment-apps/devtransitindexercerts/splunk
mv myNew* /opt/splunk/etc/deployment-apps/dev
transitindexercerts/splunk
mv myServer* /opt/splunk/etc/deployment-apps/devtransitindexer_certs/splunk

Search Head configurations

/devhfoutputs/local/server.conf

[sslConfig]
sslRootCAPath = /opt/splunk/etc/apps/devtransitforwarder_certs/splunk/myCACertificate.pem

/devhfoutputs/local/outputs.conf

[indexAndForward]
index = false

[tcpout]
defaultGroup = dev_indexers
indexAndForward = false

[tcpout:dev_indexers]
server = 10.0.0.10:9996,10.0.0.11:9996
disabled = 0

[tcpout:splunkssl]
clientCert = /opt/splunk/etc/apps/devtransitforwardercerts/splunk/myNewServerCertificate.pem
sslPassword = splunk
$certs
sslVerifyServerCert = false

Indexer configurations

/devindexersinputs/local/server.conf

[sslConfig]
sslRootCAPath = /opt/splunk/etc/apps/devtransitindexer_certs/splunk/myCACertificate.pem

/devindexersinputs/local/inputs.conf

[splunktcp-ssl:9996]
disabled = 0

[SSL]
serverCert = /opt/splunk/etc/apps/devtransitindexercerts/splunk/myNewServerCertificate.pem
sslPassword = splunk
$certs
requireClientCert= false

0 Karma
Highlighted

Re: Could I get some advice on configuring self-signed certs between Forwarder and Indexer?

Champion

on SSL areas, i got many questions as well.. i wrote this comment sometime back..
"SSL is one of the most difficult areas and its the least documented on splunk docs"

0 Karma
Highlighted

Re: Could I get some advice on configuring self-signed certs between Forwarder and Indexer?

Engager

I got this mostly working by correcting my outputs.conf. Incase it helps anyone

[tcpout]
defaultGroup = dev_indexers
indexAndForward = false
useACK = true

[tcpout:devindexers]
server = 10.0.0.10:9996,10.0.0.11:9996
disabled = 0
sslPassword = splunk
$certs
sslVerifyServerCert = true
useClientSSLCompression = true
sslCertPath = $SPLUNKHOME/etc/apps/devtransitforwardercerts/splunk/myNewServerCertificate.pem

0 Karma