I am unable to get forwarder <> indexer SSL communication to work in Amazon AWS EC2. I would appreciate any help offered as I am sure this is a somewhat dead-horse I am beating here..
I ran through with a dry-run in our lab and got SSL working correctly. I followed the exact same steps between 2 Amazon EC2 instances (same availability zone & full connectivity is working) and I get SSL certificate errors.
I've run through the steps outlined here:
These steps work in the lab.
Steps followed to create certs:
1) Create key to sign certs
openssl genrsa -des3 -out splunkCAPrivateKey.key 2048
2) Generate & Sign the CA cert
openssl req -new -key splunkCAPrivateKey.key -out splunkCACertificate.csr
openssl x509 -req -in splunkCACertificate.csr -sha1 -signkey splunkCAPrivateKey.key -CAcreateserial -out splunkCACertificate.pem -days 1095
3) Create key for server cert
openssl genrsa -des3 -out splunkServerPrivateKey.key 2048
4) Generate & Sign the Server cert
openssl req -new -key splunkServerPrivateKey.key -out splunkServerCertificate.csr
openssl x509 -req -in splunkServerCertificate.csr -sha1 -CA splunkCACertificate.pem -CAkey splunkCAPrivateKey.key -CAcreateserial -out splunkServerCertificate.pem -days 1095
cat splunkServerCertificate.pem splunkServerPrivateKey.key splunkCACertificate.pem > myNewServerCertificate.pem
Indexer Server cert folder (/opt/splunk/etc/certs/):
-rw-rw-r-- 1 splunk splunk 4.3K Mar 5 18:55 myServerCertificate.pem
-rw-rw-r-- 1 splunk splunk 1.3K Mar 5 18:55 splunkCACertificate.pem
host = st-pvc-logs
compressed = false
password = hash of password
requireClientCert = false
rootCA = /opt/splunk/etc/certs/splunkCACertificate.pem
(IPs have been changed)
Forwarder log error:
ERROR TcpOutputFd - Connection to host=10.1.1.84:9997 failed. sock_error = 0. SSL Error = error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
ls -alh /opt/splunkforwarder/etc/certs/
drwxrwxr-x 2 splunk splunk 4.0K Mar 5 11:03 .
drwxr-xr-x 13 splunk splunk 4.0K Mar 4 11:29 ..
-rw-rw-r-- 1 splunk splunk 4.3K Mar 5 11:03 myServerCertificate.pem
-rw-rw-r-- 1 splunk splunk 1.3K Mar 5 11:03 splunkCACertificate.pem
defaultGroup = default-autolb-group
server = ec2-1-2-3-4.compute-1.amazonaws.com:9997
compressed = false
sslCertPath = /opt/splunkforwarder/etc/certs/myServerCertificate.pem
sslPassword = hash of password
sslRootCAPath = /opt/splunkforwarder/etc/certs/splunkCACertificate.pem
sslVerifyServerCert = false
Indexer Server log error:
03-05-2013 21:45:51.050 +0000 ERROR TcpInputProc - Error encountered for connection from src=10.2.2.7:58189. error:14094412:SSL routines:SSL3_READ_BYTES:sslv3 alert bad certificate
I have tried creating certificates with no common name, I have tried telling splunk what common name to care about... I'm at a loss here.
I finally go this working. It's been some time but I think the issue was that I tried with multiple keys and the password hash is salted. We entered in the passphrase in plaintext in /opt/splunkforwarder/etc/system/local/outputs.conf, restarted Splunk, and it picked it up (and hashed the passphrase in the config file)...