All Apps and Add-ons

AWS Kenisis Data Stream Fails with "Could not connect to the HEC endpoint. Make sure that the certificate and the host are valid"

bentysontcxn
New Member

My splunk heavy forwarder is at https://host.domain.net:8088/ and if I send a request manually it receives data:-

curl -k "https://host.domain.net:8088/services/collector" -H "Authorization: Splunk 999a99ab-99-a '{"event": "Hello, world! Again", "sourcetype": "manual"}'
{"text":"Success","code":0}

However, if I use test data stream provided by AWS my kenisis firehose is reporting:-

2017-12-14T10:47:45.45T+0100D https://host.domain.net:8088 Could not connect to the HEC endpoint. Make sure that the certificate and the host are valid. Splunk.SSLHandshake 1

  • The SSL certificate is valid (checked with a browser(s)) and is from letsencrypt and there is a password set on the key and the password is correctly being hashed in the /local/inputs.conf

  • The machine that is running Splunk knows it’s own hostname (although that resolves to the internal IP address).

  • TCP dump is show connections from the correct source IP (and the security group is open 0/0 for testing)

10:13:08.782870 IP 34.241.197.87.35966 > 1.1.1.20.8088

All IPs, Hostnames and Keys are redacted

  • Indexer Acknowledgement is enabled (turned off to test manual posting)

  • Upgraded to 7.0.1

Platform: Ubuntu 16.04

At this point I am stumped, any bright ideas much appreciated

Labels (3)
0 Karma

zamkov
Explorer

I was having the same issue and managed to work around it by using an AWS ELB configured with an ACM certificate. I can confirm that AWS Firehose does recognize ACM generated certificates as valid. See docs at http://docs.splunk.com/Documentation/AddOns/released/Firehose/ConfigureanELB

I cannot confirm whether AWS considers lets encrypt as a valid certificate (@bentysontcxn mentioned switching to a different CA solved the issue) BUT note that your HEC may use a different certificate than the web portal. In my case, my web portal was using Let's Encrypt, so I thought I was in the same boat as @bentysontcxn, but later realized the HEC was using a self-signed certificate. To check which certificate is used by the HEC, try https://your-domain.com:8088/services/collector/health/1.0, where 8088 is the HEC listening port.

qbolbk59
Path Finder

I am using a trusted CA (Globalsign Cert) and i also validated the cert on all the URLs of HEC and it's trusted. But Kinesis is still throwing the same error that the HEC certificate is not trusted. No idea what's wrong now.

0 Karma

ayenbsc
New Member

Checking to see if you got a resolution as I have the same issue and using a valid public cert.

0 Karma

bentysontcxn
New Member

I ripped this out and started again. The problem lies here:

"The SSL certificate is valid (checked with a browser(s)) and is from letsencrypt and there is a password set on the key and the password is correctly being hashed in the /local/inputs.conf"

(See my original post)

Seems that AWS does like lets-encrypt, despite the fact that its valid in a browser(s). When I switched to a godaddy provided wildcard certificate, it works fine.

Still can't work out how to get the feed from CloudWatch into Kinesis, but that's a story for someone else's forum.

0 Karma

pauld_splunk
Splunk Employee
Splunk Employee

Amazon Kinesis Firehose requires HTTP Event Collector (HEC) endpoint to be terminated with a valid CA-signed certificate matching the DNS hostname used to connect to your HEC endpoint.

Note the requirements in the docs - "You must use a trusted CA-signed certificate. Self-signed certificates are not supported."
http://docs.splunk.com/Documentation/AddOns/released/Firehose/Hardwareandsoftwarerequirements
(you can use AWS Certificate Manager)

0 Karma

bentysontcxn
New Member

See my original post

"The SSL certificate is valid (checked with a browser(s)) and is from lets-encrypt and there is a password set on the key and the password is correctly being hashed in the /local/inputs.conf"

Not using a self signed cert and the host resolves correctly.

However see my answer, below

Kindest

0 Karma

yarick
Path Finder

@bentysontcxn : I am having exact same issue. Have you tried opening a ticket with AWS Support?

In addition to error : "Could not connect to the HEC endpoint. Make sure that the certificate and the host are valid."
I am also seeing this error a few seconds later : "An internal error occurred when attempting to deliver data. Delivery will be retried; if the error persists, it will be reported to AWS for resolution."

EDIT: ACM certificate solved the issue for me.

0 Karma
Get Updates on the Splunk Community!

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...

Thank You for Celebrating CX Day with Splunk!

Yesterday the entire team at Splunk + Cisco joined the global celebration of CX Day - celebrating our ...