- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Could I get some advice on configuring self-signed certs between Forwarder and Indexer?
Hello Everyone,
I am having trouble configuring self-signed certs and was wondering if I could possibly get some advice.
I am doing this in a test environment with the express purpose of replicating the configurations listed in the Splunk docs (.../Splunk/7.1.3/Security/Howtoself-signcertificates)
These configs are being performed on a deployment server. The deployment server's splunk.secret was replicabed to all boxes upon initial install. This splunk 7.1.2 on RHEL 7.
Currently I am getting the following error:
- ERROR TcpInputProc - Error encountered for connection from src=10.0.0.1:36014. error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
- WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read server certificate B', alert_description='unknown CA'.
Here is my configuration:
Create a key to sign your certificates.
/opt/splunk/bin/splunk cmd openssl genrsa -aes256 -out myCAPrivateKey.key 2048
splunk_$certs
Generate a new Certificate Signing Request (CSR) When prompted, create a password for the key.
/opt/splunk/bin/splunk cmd openssl req -new -key myCAPrivateKey.key -out myCACertificate.csr
splunk_$certsAnything not specified is left default/blank
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:DC
Locality Name (eg, city) []:Washington
Organization Name (eg, company) [Internet Widgits Pty Ltd]:MyTestOrg
Organizational Unit Name (eg, section) []:SecDiv
A challenge password []:splunk_$certs
Common Name (e.g. server FQDN or YOUR name) []:Deployment_ServerUse the CSR myCACertificate.csr to generate the public certificate:
/opt/splunk/bin/splunk cmd openssl x509 -req -in myCACertificate.csr -sha512 -signkey myCAPrivateKey.key -CAcreateserial -out myCACertificate.pem -days 10950
splunk_$certs
Create the server certificate for the search head to forward its data to the indexers
/opt/splunk/bin/splunk cmd openssl genrsa -aes256 -out myServerPrivateKey.key 2048
splunk_$certs
- Generate and sign a new server certificate
/opt/splunk/bin/splunk cmd openssl req -new -key myServerPrivateKey.key -out myServerCertificate.csr
splunk_$certs
- Anything not specified is left default/blank Country Name (2 letter code) [AU]:US State or Province Name (full name) [Some-State]:DC Locality Name (eg, city) []:Washington Organization Name (eg, company) [Internet Widgits Pty Ltd]:MyTestOrg Organizational Unit Name (eg, section) []:SecDiv Common Name (e.g. server FQDN or YOUR name) []:Search_Head A challenge password []:splunk_$certs
/opt/splunk/bin/splunk cmd openssl x509 -req -in myServerCertificate.csr -SHA256 -CA myCACertificate.pem -CAkey myCAPrivateKey.key -CAcreateserial -out myServerCertificate.pem -days 1095
splunk_$certs
- Create a single PEM file
- Once you have your certificates, you must combine the server certificate and your keys into a single file that Splunk software can use.
cat myServerCertificate.pem myServerPrivateKey.key myCACertificate.pem > myNewServerCertificate.pem
- The CA cert is copied to a deployment app so it can be reused. The Search_Head/server certs are moved.
- The 'dev_transit_forwarder_certs' app is transfered to the search head via the deployment server
cp myCA* /opt/splunk/etc/deployment-apps/dev_transit_forwarder_certs/splunk
mv myNew* /opt/splunk/etc/deployment-apps/dev_transit_forwarder_certs/splunk
mv myServer* /opt/splunk/etc/deployment-apps/dev_transit_forwarder_certs/splunk
Create the server certificate for the search head to forward its data to the indexers
/opt/splunk/bin/splunk cmd openssl genrsa -aes256 -out myServerPrivateKey.key 2048
splunk_$certs
- Generate and sign a new server certificate
/opt/splunk/bin/splunk cmd openssl req -new -key myServerPrivateKey.key -out myServerCertificate.csr
splunk_$certs
- Anything not specified is left default/blank Country Name (2 letter code) [AU]:US State or Province Name (full name) [Some-State]:DC Locality Name (eg, city) []:Washington Organization Name (eg, company) [Internet Widgits Pty Ltd]:MyTestOrg Organizational Unit Name (eg, section) []:SecDiv Common Name (e.g. server FQDN or YOUR name) []:Indexer A challenge password []:splunk_$certs
/opt/splunk/bin/splunk cmd openssl x509 -req -in myServerCertificate.csr -SHA256 -CA myCACertificate.pem -CAkey myCAPrivateKey.key -CAcreateserial -out myServerCertificate.pem -days 1095
splunk_$certs
- Create a single PEM file
- Once you have your certificates, you must combine the server certificate and your keys into a single file that Splunk software can use.
cat myServerCertificate.pem myServerPrivateKey.key myCACertificate.pem > myNewServerCertificate.pem
- The certs are copied to a deployment app.
- The 'dev_transit_indexer_certs' app is transfered to two indexers via the deployment server
cp myCA* /opt/splunk/etc/deployment-apps/dev_transit_indexer_certs/splunk
mv myNew* /opt/splunk/etc/deployment-apps/dev_transit_indexer_certs/splunk
mv myServer* /opt/splunk/etc/deployment-apps/dev_transit_indexer_certs/splunk
Search Head configurations
/dev_hf_outputs/local/server.conf
[sslConfig]
sslRootCAPath = /opt/splunk/etc/apps/dev_transit_forwarder_certs/splunk/myCACertificate.pem
/dev_hf_outputs/local/outputs.conf
[indexAndForward]
index = false
[tcpout]
defaultGroup = dev_indexers
indexAndForward = false
[tcpout:dev_indexers]
server = 10.0.0.10:9996,10.0.0.11:9996
disabled = 0
[tcpout:splunkssl]
clientCert = /opt/splunk/etc/apps/dev_transit_forwarder_certs/splunk/myNewServerCertificate.pem
sslPassword = splunk_$certs
sslVerifyServerCert = false
Indexer configurations
/dev_indexers_inputs/local/server.conf
[sslConfig]
sslRootCAPath = /opt/splunk/etc/apps/dev_transit_indexer_certs/splunk/myCACertificate.pem
/dev_indexers_inputs/local/inputs.conf
[splunktcp-ssl:9996]
disabled = 0
[SSL]
serverCert = /opt/splunk/etc/apps/dev_transit_indexer_certs/splunk/myNewServerCertificate.pem
sslPassword = splunk_$certs
requireClientCert= false
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I got this mostly working by correcting my outputs.conf. Incase it helps anyone
[tcpout]
defaultGroup = dev_indexers
indexAndForward = false
useACK = true
[tcpout:dev_indexers]
server = 10.0.0.10:9996,10.0.0.11:9996
disabled = 0
sslPassword = splunk_$certs
sslVerifyServerCert = true
useClientSSLCompression = true
sslCertPath = $SPLUNK_HOME/etc/apps/dev_transit_forwarder_certs/splunk/myNewServerCertificate.pem
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

on SSL areas, i got many questions as well.. i wrote this comment sometime back..
"SSL is one of the most difficult areas and its the least documented on splunk docs"
Sekar
PS - If this or any post helped you in any way, pls consider upvoting, thanks for reading !
