My splunk heavy forwarder is at https://host.domain.net:8088/ and if I send a request manually it receives data:-
curl -k "https://host.domain.net:8088/services/collector" -H "Authorization: Splunk 999a99ab-99-a '{"event": "Hello, world! Again", "sourcetype": "manual"}'
{"text":"Success","code":0}
However, if I use test data stream provided by AWS my kenisis firehose is reporting:-
2017-12-14T10:47:45.45T+0100D https://host.domain.net:8088 Could not connect to the HEC endpoint. Make sure that the certificate and the host are valid. Splunk.SSLHandshake 1
The SSL certificate is valid (checked with a browser(s)) and is from letsencrypt and there is a password set on the key and the password is correctly being hashed in the /local/inputs.conf
The machine that is running Splunk knows it’s own hostname (although that resolves to the internal IP address).
TCP dump is show connections from the correct source IP (and the security group is open 0/0 for testing)
10:13:08.782870 IP 34.241.197.87.35966 > 1.1.1.20.8088
All IPs, Hostnames and Keys are redacted
Indexer Acknowledgement is enabled (turned off to test manual posting)
Upgraded to 7.0.1
Platform: Ubuntu 16.04
At this point I am stumped, any bright ideas much appreciated
I was having the same issue and managed to work around it by using an AWS ELB configured with an ACM certificate. I can confirm that AWS Firehose does recognize ACM generated certificates as valid. See docs at http://docs.splunk.com/Documentation/AddOns/released/Firehose/ConfigureanELB
I cannot confirm whether AWS considers lets encrypt as a valid certificate (@bentysontcxn mentioned switching to a different CA solved the issue) BUT note that your HEC may use a different certificate than the web portal. In my case, my web portal was using Let's Encrypt, so I thought I was in the same boat as @bentysontcxn, but later realized the HEC was using a self-signed certificate. To check which certificate is used by the HEC, try https://your-domain.com:8088/services/collector/health/1.0, where 8088 is the HEC listening port.
I am using a trusted CA (Globalsign Cert) and i also validated the cert on all the URLs of HEC and it's trusted. But Kinesis is still throwing the same error that the HEC certificate is not trusted. No idea what's wrong now.
Checking to see if you got a resolution as I have the same issue and using a valid public cert.
I ripped this out and started again. The problem lies here:
"The SSL certificate is valid (checked with a browser(s)) and is from letsencrypt and there is a password set on the key and the password is correctly being hashed in the /local/inputs.conf"
(See my original post)
Seems that AWS does like lets-encrypt, despite the fact that its valid in a browser(s). When I switched to a godaddy provided wildcard certificate, it works fine.
Still can't work out how to get the feed from CloudWatch into Kinesis, but that's a story for someone else's forum.
Amazon Kinesis Firehose requires HTTP Event Collector (HEC) endpoint to be terminated with a valid CA-signed certificate matching the DNS hostname used to connect to your HEC endpoint.
Note the requirements in the docs - "You must use a trusted CA-signed certificate. Self-signed certificates are not supported."
http://docs.splunk.com/Documentation/AddOns/released/Firehose/Hardwareandsoftwarerequirements
(you can use AWS Certificate Manager)
See my original post
"The SSL certificate is valid (checked with a browser(s)) and is from lets-encrypt and there is a password set on the key and the password is correctly being hashed in the /local/inputs.conf"
Not using a self signed cert and the host resolves correctly.
However see my answer, below
Kindest
@bentysontcxn : I am having exact same issue. Have you tried opening a ticket with AWS Support?
In addition to error : "Could not connect to the HEC endpoint. Make sure that the certificate and the host are valid."
I am also seeing this error a few seconds later : "An internal error occurred when attempting to deliver data. Delivery will be retried; if the error persists, it will be reported to AWS for resolution."
EDIT: ACM certificate solved the issue for me.