All Apps and Add-ons
Highlighted

How come my AWS Kinesis Firehose is failing to connect to HEC due to SSL Handshake?

Communicator

Has anyone successfully achieved Kinesis Firehose to a HEC secured with letsencrypt certs?

I've used letsencrypt to generate SSL certs for my Splunk server. I've used those in web.conf to secure Splunk web, and I'm trying to use them with HEC to permit SSL connections.

I've got as far as cURL submitting events using HTTPS (without -k to ignore cert errors!) and if I browse to https://splunk.mydomain.com:8088, my browser is happy with the cert.

The problem is that AWS Kinesis Firehose isn't happy.

Cloudwatch is reporting

{
    "deliveryStreamARN": "arn:aws:firehose:us-west-2:123455522430:deliverystream/my-delivery-stream",
    "destination": "https://splunk.mydomain.com:8088",
    "deliveryStreamVersionId": 1,
    "message": "Could not connect to the HEC endpoint. Make sure that the certificate and the host are valid.",
    "errorCode": "Splunk.SSLHandshake"
}

web.conf looks like this:

[settings]
enableSplunkWebSSL = true
privKeyPath = etc/auth/mydomain.com/privkey.pem
caCertPath = etc/auth/mydomain.com/fullchain.pem

/opt/splunk/etc/apps/splunk_httpinput/local/inputs.conf currently looks like this

[http]
disabled = 0
index = temp
enableSSL = 1
sslVersions = *,-ssl2
allowSslCompression = true
allowSslRenegotiation = true
caCertFile =  /opt/splunk/etc/auth/mydomain.com/cert.pem
sslKeysfile = /opt/splunk/etc/auth/mydomain.com/privkey.pem
sslKeysfilePassword =
ackIdleCleanup = true

I've tried using openssl to produce a password-protected key file (privkey.pem) and specify sslKeysfilePassword — no difference.

I read one answers post saying that letsencrypt wasn't trusted by AWS, but that seemed unlikely. All help appreciated!

I'm using Splunk Enterprise 7.2.4.

Highlighted

Re: How come my AWS Kinesis Firehose is failing to connect to HEC due to SSL Handshake?

Communicator

Similar to other threads on this, I managed to get Kinesis Firehose -> HEC working by doing the following:

  • Standup a splunk server in AWS
  • Configure it to listen with HEC and disable HTTPS
  • Put an ELB in front of it, listening for HTTPS on 8088 and forwarding to HTTP (not S) on 8088
  • Create a DNS CNAME record for splunk.mydomain.com -> myloadbalancerid.elb.amazonaws.com
  • Use AWS ACM to issue a cert for that name and associate it with the ELB
  • Create a Firehose data stream sending data to https://splunk.mydomain.com:8088

It's frustrating to not know why Firehose wasn't happy sending to my original HEC - potentially due to LetsEncrypt being the CA but that's just speculation.

View solution in original post

Highlighted

Re: How come my AWS Kinesis Firehose is failing to connect to HEC due to SSL Handshake?

New Member

Hey @gf13579 did you setup an internal or external elb for this?
Also what did you add to your elb security group?

0 Karma
Highlighted

Re: How come my AWS Kinesis Firehose is failing to connect to HEC due to SSL Handshake?

Communicator

Hi @obla. I setup an external (scheme: internet-facing) one though, assuming you can issue certs from ACM to internal ELBs, internal should be fine.

The ELB's SG contains inbound rules for TCP 8088 and 443 from 0.0.0.0/0 though it should just need 8088 or whatever port you configure FireHose to send on. How you restrict inbound traffic to the FireHose is beyond me at this point.

Outbound is unrestricted but could've been limited to 8088 to the security group of the EC2 VM hosting splunk.

Not a very locked-down setup but it was for a temporary(...) solution.

0 Karma