Self Signed SSL Certificate problem in Amazon EC2 (works in lab)

[JoshSupport]

I am unable to get forwarder <> indexer SSL communication to work in Amazon AWS EC2. I would appreciate any help offered as I am sure this is a somewhat dead-horse I am beating here..

I ran through with a dry-run in our lab and got SSL working correctly. I followed the exact same steps between 2 Amazon EC2 instances (same availability zone & full connectivity is working) and I get SSL certificate errors.

---

I've run through the steps outlined here:
http://splunk-base.splunk.com/answers/7164/how-do-i-set-up-ssl-forwarding-with-new-self-signed-certi...
These steps work in the lab.

Steps followed to create certs:

1) Create key to sign certs
openssl genrsa -des3 -out splunkCAPrivateKey.key 2048

2) Generate & Sign the CA cert
openssl req -new -key splunkCAPrivateKey.key -out splunkCACertificate.csr
openssl x509 -req -in splunkCACertificate.csr -sha1 -signkey splunkCAPrivateKey.key -CAcreateserial -out splunkCACertificate.pem -days 1095

3) Create key for server cert
openssl genrsa -des3 -out splunkServerPrivateKey.key 2048

4) Generate & Sign the Server cert
openssl req -new -key splunkServerPrivateKey.key -out splunkServerCertificate.csr
openssl x509 -req -in splunkServerCertificate.csr -sha1 -CA splunkCACertificate.pem -CAkey splunkCAPrivateKey.key -CAcreateserial -out splunkServerCertificate.pem -days 1095
cat splunkServerCertificate.pem splunkServerPrivateKey.key splunkCACertificate.pem > myNewServerCertificate.pem

Indexer Server cert folder (/opt/splunk/etc/certs/):
-rw-rw-r-- 1 splunk splunk 4.3K Mar 5 18:55 myServerCertificate.pem
-rw-rw-r-- 1 splunk splunk 1.3K Mar 5 18:55 splunkCACertificate.pem

Indexer Server inputs config (/opt/splunk/etc/system/local/inputs.conf):

[default]
host = st-pvc-logs

[splunktcp-ssl:9997]
compressed = false

[SSL]
password = hash of password
requireClientCert = false
rootCA = /opt/splunk/etc/certs/splunkCACertificate.pem

serverCert = /opt/splunk/etc/certs/myServerCertificate.pem

PROBLEM:

(IPs have been changed)

Forwarder log error:
ERROR TcpOutputFd - Connection to host=10.1.1.84:9997 failed. sock_error = 0. SSL Error = error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

ls -alh /opt/splunkforwarder/etc/certs/
total 20K
drwxrwxr-x 2 splunk splunk 4.0K Mar 5 11:03 .
drwxr-xr-x 13 splunk splunk 4.0K Mar 4 11:29 ..
-rw-rw-r-- 1 splunk splunk 4.3K Mar 5 11:03 myServerCertificate.pem
-rw-rw-r-- 1 splunk splunk 1.3K Mar 5 11:03 splunkCACertificate.pem

Forwarder outputs config (/opt/splunkforwarder/etc/system/local/outputs.conf):

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = ec2-1-2-3-4.compute-1.amazonaws.com:9997

[tcpout-server://ec2-1-2-3-4.compute-1.amazonaws.com:9997]
compressed = false
sslCertPath = /opt/splunkforwarder/etc/certs/myServerCertificate.pem
sslPassword = hash of password
sslRootCAPath = /opt/splunkforwarder/etc/certs/splunkCACertificate.pem
sslVerifyServerCert = false

sslCommonNameToCheck = mydomain.com

Indexer Server log error:
03-05-2013 21:45:51.050 +0000 ERROR TcpInputProc - Error encountered for connection from src=10.2.2.7:58189. error:14094412:SSL routines:SSL3_READ_BYTES:sslv3 alert bad certificate

I have tried creating certificates with no common name, I have tried telling splunk what common name to care about... I'm at a loss here.

[JoshSupport]

I finally go this working. It's been some time but I think the issue was that I tried with multiple keys and the password hash is salted. We entered in the passphrase in plaintext in /opt/splunkforwarder/etc/system/local/outputs.conf, restarted Splunk, and it picked it up (and hashed the passphrase in the config file)...

[jmahosky]

Did you find a solution? I am running into the exact same issue.