Security

Could I get some advice on configuring self-signed certs between Forwarder and Indexer?

liquidclay23
Explorer

Hello Everyone,

I am having trouble configuring self-signed certs and was wondering if I could possibly get some advice.

I am doing this in a test environment with the express purpose of replicating the configurations listed in the Splunk docs (.../Splunk/7.1.3/Security/Howtoself-signcertificates)

These configs are being performed on a deployment server. The deployment server's splunk.secret was replicabed to all boxes upon initial install. This splunk 7.1.2 on RHEL 7.

Currently I am getting the following error:
- ERROR TcpInputProc - Error encountered for connection from src=10.0.0.1:36014. error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
- WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read server certificate B', alert_description='unknown CA'.

Here is my configuration:

Create a key to sign your certificates.

/opt/splunk/bin/splunk cmd openssl genrsa -aes256 -out myCAPrivateKey.key 2048
splunk_$certs

  • Generate a new Certificate Signing Request (CSR) When prompted, create a password for the key.
    /opt/splunk/bin/splunk cmd openssl req -new -key myCAPrivateKey.key -out myCACertificate.csr
    splunk_$certs

  • Anything not specified is left default/blank
    Country Name (2 letter code) [AU]:US
    State or Province Name (full name) [Some-State]:DC
    Locality Name (eg, city) []:Washington
    Organization Name (eg, company) [Internet Widgits Pty Ltd]:MyTestOrg
    Organizational Unit Name (eg, section) []:SecDiv
    A challenge password []:splunk_$certs
    Common Name (e.g. server FQDN or YOUR name) []:Deployment_Server

  • Use the CSR myCACertificate.csr to generate the public certificate:
    /opt/splunk/bin/splunk cmd openssl x509 -req -in myCACertificate.csr -sha512 -signkey myCAPrivateKey.key -CAcreateserial -out myCACertificate.pem -days 10950
    splunk_$certs

Create the server certificate for the search head to forward its data to the indexers

/opt/splunk/bin/splunk cmd openssl genrsa -aes256 -out myServerPrivateKey.key 2048
splunk_$certs

  • Generate and sign a new server certificate

/opt/splunk/bin/splunk cmd openssl req -new -key myServerPrivateKey.key -out myServerCertificate.csr
splunk_$certs

  • Anything not specified is left default/blank Country Name (2 letter code) [AU]:US State or Province Name (full name) [Some-State]:DC Locality Name (eg, city) []:Washington Organization Name (eg, company) [Internet Widgits Pty Ltd]:MyTestOrg Organizational Unit Name (eg, section) []:SecDiv Common Name (e.g. server FQDN or YOUR name) []:Search_Head A challenge password []:splunk_$certs

/opt/splunk/bin/splunk cmd openssl x509 -req -in myServerCertificate.csr -SHA256 -CA myCACertificate.pem -CAkey myCAPrivateKey.key -CAcreateserial -out myServerCertificate.pem -days 1095
splunk_$certs

  • Create a single PEM file
  • Once you have your certificates, you must combine the server certificate and your keys into a single file that Splunk software can use.

cat myServerCertificate.pem myServerPrivateKey.key myCACertificate.pem > myNewServerCertificate.pem

  • The CA cert is copied to a deployment app so it can be reused. The Search_Head/server certs are moved.
  • The 'dev_transit_forwarder_certs' app is transfered to the search head via the deployment server

cp myCA* /opt/splunk/etc/deployment-apps/dev_transit_forwarder_certs/splunk
mv myNew* /opt/splunk/etc/deployment-apps/dev_transit_forwarder_certs/splunk
mv myServer* /opt/splunk/etc/deployment-apps/dev_transit_forwarder_certs/splunk

Create the server certificate for the search head to forward its data to the indexers

/opt/splunk/bin/splunk cmd openssl genrsa -aes256 -out myServerPrivateKey.key 2048
splunk_$certs

  • Generate and sign a new server certificate

/opt/splunk/bin/splunk cmd openssl req -new -key myServerPrivateKey.key -out myServerCertificate.csr
splunk_$certs

  • Anything not specified is left default/blank Country Name (2 letter code) [AU]:US State or Province Name (full name) [Some-State]:DC Locality Name (eg, city) []:Washington Organization Name (eg, company) [Internet Widgits Pty Ltd]:MyTestOrg Organizational Unit Name (eg, section) []:SecDiv Common Name (e.g. server FQDN or YOUR name) []:Indexer A challenge password []:splunk_$certs

/opt/splunk/bin/splunk cmd openssl x509 -req -in myServerCertificate.csr -SHA256 -CA myCACertificate.pem -CAkey myCAPrivateKey.key -CAcreateserial -out myServerCertificate.pem -days 1095
splunk_$certs

  • Create a single PEM file
  • Once you have your certificates, you must combine the server certificate and your keys into a single file that Splunk software can use.

cat myServerCertificate.pem myServerPrivateKey.key myCACertificate.pem > myNewServerCertificate.pem

  • The certs are copied to a deployment app.
  • The 'dev_transit_indexer_certs' app is transfered to two indexers via the deployment server

cp myCA* /opt/splunk/etc/deployment-apps/dev_transit_indexer_certs/splunk
mv myNew* /opt/splunk/etc/deployment-apps/dev_transit_indexer_certs/splunk
mv myServer* /opt/splunk/etc/deployment-apps/dev_transit_indexer_certs/splunk

Search Head configurations

/dev_hf_outputs/local/server.conf

[sslConfig]
sslRootCAPath = /opt/splunk/etc/apps/dev_transit_forwarder_certs/splunk/myCACertificate.pem

/dev_hf_outputs/local/outputs.conf

[indexAndForward]
index = false

[tcpout]
defaultGroup = dev_indexers
indexAndForward = false

[tcpout:dev_indexers]
server = 10.0.0.10:9996,10.0.0.11:9996
disabled = 0

[tcpout:splunkssl]
clientCert = /opt/splunk/etc/apps/dev_transit_forwarder_certs/splunk/myNewServerCertificate.pem
sslPassword = splunk_$certs
sslVerifyServerCert = false

Indexer configurations

/dev_indexers_inputs/local/server.conf

[sslConfig]
sslRootCAPath = /opt/splunk/etc/apps/dev_transit_indexer_certs/splunk/myCACertificate.pem

/dev_indexers_inputs/local/inputs.conf

[splunktcp-ssl:9996]
disabled = 0

[SSL]
serverCert = /opt/splunk/etc/apps/dev_transit_indexer_certs/splunk/myNewServerCertificate.pem
sslPassword = splunk_$certs
requireClientCert= false

0 Karma

liquidclay23
Explorer

I got this mostly working by correcting my outputs.conf. Incase it helps anyone

[tcpout]
defaultGroup = dev_indexers
indexAndForward = false
useACK = true

[tcpout:dev_indexers]
server = 10.0.0.10:9996,10.0.0.11:9996
disabled = 0
sslPassword = splunk_$certs
sslVerifyServerCert = true
useClientSSLCompression = true
sslCertPath = $SPLUNK_HOME/etc/apps/dev_transit_forwarder_certs/splunk/myNewServerCertificate.pem

0 Karma

inventsekar
Ultra Champion

on SSL areas, i got many questions as well.. i wrote this comment sometime back..
"SSL is one of the most difficult areas and its the least documented on splunk docs"

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...