All Apps and Add-ons

How come my AWS Kinesis Firehose is failing to connect to HEC due to SSL Handshake?

gf13579
Communicator

Has anyone successfully achieved Kinesis Firehose to a HEC secured with letsencrypt certs?

I've used letsencrypt to generate SSL certs for my Splunk server. I've used those in web.conf to secure Splunk web, and I'm trying to use them with HEC to permit SSL connections.

I've got as far as cURL submitting events using HTTPS (without -k to ignore cert errors!) and if I browse to https://splunk.mydomain.com:8088, my browser is happy with the cert.

The problem is that AWS Kinesis Firehose isn't happy.

Cloudwatch is reporting

{
    "deliveryStreamARN": "arn:aws:firehose:us-west-2:123455522430:deliverystream/my-delivery-stream",
    "destination": "https://splunk.mydomain.com:8088",
    "deliveryStreamVersionId": 1,
    "message": "Could not connect to the HEC endpoint. Make sure that the certificate and the host are valid.",
    "errorCode": "Splunk.SSLHandshake"
}

web.conf looks like this:

[settings]
enableSplunkWebSSL = true
privKeyPath = etc/auth/mydomain.com/privkey.pem
caCertPath = etc/auth/mydomain.com/fullchain.pem

/opt/splunk/etc/apps/splunk_httpinput/local/inputs.conf currently looks like this

[http]
disabled = 0
index = temp
enableSSL = 1
sslVersions = *,-ssl2
allowSslCompression = true
allowSslRenegotiation = true
caCertFile =  /opt/splunk/etc/auth/mydomain.com/cert.pem
sslKeysfile = /opt/splunk/etc/auth/mydomain.com/privkey.pem
sslKeysfilePassword =
ackIdleCleanup = true

I've tried using openssl to produce a password-protected key file (privkey.pem) and specify sslKeysfilePassword — no difference.

I read one answers post saying that letsencrypt wasn't trusted by AWS, but that seemed unlikely. All help appreciated!

I'm using Splunk Enterprise 7.2.4.

1 Solution

gf13579
Communicator

Similar to other threads on this, I managed to get Kinesis Firehose -> HEC working by doing the following:

  • Standup a splunk server in AWS
  • Configure it to listen with HEC and disable HTTPS
  • Put an ELB in front of it, listening for HTTPS on 8088 and forwarding to HTTP (not S) on 8088
  • Create a DNS CNAME record for splunk.mydomain.com -> myloadbalancerid.elb.amazonaws.com
  • Use AWS ACM to issue a cert for that name and associate it with the ELB
  • Create a Firehose data stream sending data to https://splunk.mydomain.com:8088

It's frustrating to not know why Firehose wasn't happy sending to my original HEC - potentially due to LetsEncrypt being the CA but that's just speculation.

View solution in original post

gf13579
Communicator

Similar to other threads on this, I managed to get Kinesis Firehose -> HEC working by doing the following:

  • Standup a splunk server in AWS
  • Configure it to listen with HEC and disable HTTPS
  • Put an ELB in front of it, listening for HTTPS on 8088 and forwarding to HTTP (not S) on 8088
  • Create a DNS CNAME record for splunk.mydomain.com -> myloadbalancerid.elb.amazonaws.com
  • Use AWS ACM to issue a cert for that name and associate it with the ELB
  • Create a Firehose data stream sending data to https://splunk.mydomain.com:8088

It's frustrating to not know why Firehose wasn't happy sending to my original HEC - potentially due to LetsEncrypt being the CA but that's just speculation.

View solution in original post

TheKennyD
Observer

Did you ever figure out how to make the HEC use a trusted SSL certificate instead of the default self-signed cert? 

0 Karma

obla
New Member

Hey @gf13579 did you setup an internal or external elb for this?
Also what did you add to your elb security group?

0 Karma

gf13579
Communicator

Hi @obla. I setup an external (scheme: internet-facing) one though, assuming you can issue certs from ACM to internal ELBs, internal should be fine.

The ELB's SG contains inbound rules for TCP 8088 and 443 from 0.0.0.0/0 though it should just need 8088 or whatever port you configure FireHose to send on. How you restrict inbound traffic to the FireHose is beyond me at this point.

Outbound is unrestricted but could've been limited to 8088 to the security group of the EC2 VM hosting splunk.

Not a very locked-down setup but it was for a temporary(...) solution.

0 Karma

TheKennyD
Observer

Hello, 

Is there really no way to get Splunk's HEC to answer SSL requests using a trusted CA certificate?

I used to use a solution to send AWS Security Hub data to Splunk's HEC via a HTTP. Then along came TRUMPET which requires a trusted SSL certificate and will not used Splunk's self signed certificate. So I went out and purchased a self signed cert, then configured the Splunk web server to use it. However, the HEC is still using Splunk's self signed certificate. I've looked at several attempts online to configure server.conf and input.conf to point to that certificate but none of those have worked. 

Does anyone know how to configure Splunk's HEC to use SSL configured with a trusted certificate? 

Thank you, 

Ken 

 

 

 

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!