Deployment Architecture

Self Signed SSL Certificate problem in Amazon EC2 (works in lab)

JoshSupport
Engager

I am unable to get forwarder <> indexer SSL communication to work in Amazon AWS EC2. I would appreciate any help offered as I am sure this is a somewhat dead-horse I am beating here..

I ran through with a dry-run in our lab and got SSL working correctly. I followed the exact same steps between 2 Amazon EC2 instances (same availability zone & full connectivity is working) and I get SSL certificate errors.


I've run through the steps outlined here:
http://splunk-base.splunk.com/answers/7164/how-do-i-set-up-ssl-forwarding-with-new-self-signed-certi...
These steps work in the lab.

Steps followed to create certs:

1) Create key to sign certs
openssl genrsa -des3 -out splunkCAPrivateKey.key 2048

2) Generate & Sign the CA cert
openssl req -new -key splunkCAPrivateKey.key -out splunkCACertificate.csr
openssl x509 -req -in splunkCACertificate.csr -sha1 -signkey splunkCAPrivateKey.key -CAcreateserial -out splunkCACertificate.pem -days 1095

3) Create key for server cert
openssl genrsa -des3 -out splunkServerPrivateKey.key 2048

4) Generate & Sign the Server cert
openssl req -new -key splunkServerPrivateKey.key -out splunkServerCertificate.csr
openssl x509 -req -in splunkServerCertificate.csr -sha1 -CA splunkCACertificate.pem -CAkey splunkCAPrivateKey.key -CAcreateserial -out splunkServerCertificate.pem -days 1095
cat splunkServerCertificate.pem splunkServerPrivateKey.key splunkCACertificate.pem > myNewServerCertificate.pem

Indexer Server cert folder (/opt/splunk/etc/certs/):
-rw-rw-r-- 1 splunk splunk 4.3K Mar 5 18:55 myServerCertificate.pem
-rw-rw-r-- 1 splunk splunk 1.3K Mar 5 18:55 splunkCACertificate.pem

Indexer Server inputs config (/opt/splunk/etc/system/local/inputs.conf):

[default]
host = st-pvc-logs

[splunktcp-ssl:9997]
compressed = false

[SSL]
password = hash of password
requireClientCert = false
rootCA = /opt/splunk/etc/certs/splunkCACertificate.pem

serverCert = /opt/splunk/etc/certs/myServerCertificate.pem

PROBLEM:

(IPs have been changed)

Forwarder log error:
ERROR TcpOutputFd - Connection to host=10.1.1.84:9997 failed. sock_error = 0. SSL Error = error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

ls -alh /opt/splunkforwarder/etc/certs/
total 20K
drwxrwxr-x 2 splunk splunk 4.0K Mar 5 11:03 .
drwxr-xr-x 13 splunk splunk 4.0K Mar 4 11:29 ..
-rw-rw-r-- 1 splunk splunk 4.3K Mar 5 11:03 myServerCertificate.pem
-rw-rw-r-- 1 splunk splunk 1.3K Mar 5 11:03 splunkCACertificate.pem

Forwarder outputs config (/opt/splunkforwarder/etc/system/local/outputs.conf):

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = ec2-1-2-3-4.compute-1.amazonaws.com:9997

[tcpout-server://ec2-1-2-3-4.compute-1.amazonaws.com:9997]
compressed = false
sslCertPath = /opt/splunkforwarder/etc/certs/myServerCertificate.pem
sslPassword = hash of password
sslRootCAPath = /opt/splunkforwarder/etc/certs/splunkCACertificate.pem
sslVerifyServerCert = false

sslCommonNameToCheck = mydomain.com

Indexer Server log error:
03-05-2013 21:45:51.050 +0000 ERROR TcpInputProc - Error encountered for connection from src=10.2.2.7:58189. error:14094412:SSL routines:SSL3_READ_BYTES:sslv3 alert bad certificate

I have tried creating certificates with no common name, I have tried telling splunk what common name to care about... I'm at a loss here.

Tags (3)

JoshSupport
Engager

I finally go this working. It's been some time but I think the issue was that I tried with multiple keys and the password hash is salted. We entered in the passphrase in plaintext in /opt/splunkforwarder/etc/system/local/outputs.conf, restarted Splunk, and it picked it up (and hashed the passphrase in the config file)...

jmahosky
Engager

Did you find a solution? I am running into the exact same issue.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...