Deployment Architecture

Self Signed SSL Certificate problem in Amazon EC2 (works in lab)


I am unable to get forwarder <> indexer SSL communication to work in Amazon AWS EC2. I would appreciate any help offered as I am sure this is a somewhat dead-horse I am beating here..

I ran through with a dry-run in our lab and got SSL working correctly. I followed the exact same steps between 2 Amazon EC2 instances (same availability zone & full connectivity is working) and I get SSL certificate errors.

I've run through the steps outlined here:
These steps work in the lab.

Steps followed to create certs:

1) Create key to sign certs
openssl genrsa -des3 -out splunkCAPrivateKey.key 2048

2) Generate & Sign the CA cert
openssl req -new -key splunkCAPrivateKey.key -out splunkCACertificate.csr
openssl x509 -req -in splunkCACertificate.csr -sha1 -signkey splunkCAPrivateKey.key -CAcreateserial -out splunkCACertificate.pem -days 1095

3) Create key for server cert
openssl genrsa -des3 -out splunkServerPrivateKey.key 2048

4) Generate & Sign the Server cert
openssl req -new -key splunkServerPrivateKey.key -out splunkServerCertificate.csr
openssl x509 -req -in splunkServerCertificate.csr -sha1 -CA splunkCACertificate.pem -CAkey splunkCAPrivateKey.key -CAcreateserial -out splunkServerCertificate.pem -days 1095
cat splunkServerCertificate.pem splunkServerPrivateKey.key splunkCACertificate.pem > myNewServerCertificate.pem

Indexer Server cert folder (/opt/splunk/etc/certs/):
-rw-rw-r-- 1 splunk splunk 4.3K Mar 5 18:55 myServerCertificate.pem
-rw-rw-r-- 1 splunk splunk 1.3K Mar 5 18:55 splunkCACertificate.pem

Indexer Server inputs config (/opt/splunk/etc/system/local/inputs.conf):

host = st-pvc-logs

compressed = false

password = hash of password
requireClientCert = false
rootCA = /opt/splunk/etc/certs/splunkCACertificate.pem

serverCert = /opt/splunk/etc/certs/myServerCertificate.pem


(IPs have been changed)

Forwarder log error:
ERROR TcpOutputFd - Connection to host= failed. sock_error = 0. SSL Error = error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

ls -alh /opt/splunkforwarder/etc/certs/
total 20K
drwxrwxr-x 2 splunk splunk 4.0K Mar 5 11:03 .
drwxr-xr-x 13 splunk splunk 4.0K Mar 4 11:29 ..
-rw-rw-r-- 1 splunk splunk 4.3K Mar 5 11:03 myServerCertificate.pem
-rw-rw-r-- 1 splunk splunk 1.3K Mar 5 11:03 splunkCACertificate.pem

Forwarder outputs config (/opt/splunkforwarder/etc/system/local/outputs.conf):

defaultGroup = default-autolb-group

server =

compressed = false
sslCertPath = /opt/splunkforwarder/etc/certs/myServerCertificate.pem
sslPassword = hash of password
sslRootCAPath = /opt/splunkforwarder/etc/certs/splunkCACertificate.pem
sslVerifyServerCert = false

sslCommonNameToCheck =

Indexer Server log error:
03-05-2013 21:45:51.050 +0000 ERROR TcpInputProc - Error encountered for connection from src= error:14094412:SSL routines:SSL3_READ_BYTES:sslv3 alert bad certificate

I have tried creating certificates with no common name, I have tried telling splunk what common name to care about... I'm at a loss here.

Tags (3)


I finally go this working. It's been some time but I think the issue was that I tried with multiple keys and the password hash is salted. We entered in the passphrase in plaintext in /opt/splunkforwarder/etc/system/local/outputs.conf, restarted Splunk, and it picked it up (and hashed the passphrase in the config file)...


Did you find a solution? I am running into the exact same issue.

0 Karma
Get Updates on the Splunk Community!

Splunk Security Content for Threat Detection & Response, Q1 Roundup

Join Principal Threat Researcher, Michael Haag, as he walks through:An introduction to the Splunk Threat ...

Splunk Life | Happy Pride Month!

Happy Pride Month, Splunk Community! &#x1f308; In the United States, as well as many countries around the ...

SplunkTrust | Where Are They Now - Michael Uschmann

The Background Five years ago, Splunk published several videos showcasing members of the SplunkTrust to share ...