Splunk HEC Exporter failing: tls: failed to verify certificate: x509

[AntonioJimenez]

Following the documentation https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector#Send_data_to_HTTP_... 

• I have:
  • Created a trial account in Splunk Cloud platform
  • Generated a HEC Token
  • Send telemetry data to Splunk Cloud platform using a OpenTelemetry collectory with Splunk HEC exporter 

splunk_hec:
  token: "<hec-token>"
  endpoint: https://prd-p-e7xnh.splunkcloud.com:8088/services/collector/event
  source: "otel"
  sourcetype: "otel"
  splunk_app_name: "ThousandEyes OpenTelemetry"
  tls:
    insecure: false

I see the following error in my `otel-collector`:

Post "https://splunkcloud.com:8088/services/collector/event": tls: failed to verify certificate: x509: certificate is not valid for any names, but wanted to match splunkcloud.com

The endpoint `https://prd-p-e7xnh.splunkcloud.com:8088` seems to have a invalid certificate. It was sign by a self-sign CA. It does not include subject name for the endpoint.

openssl s_client -showcerts -connect prd-p-e7xnh.splunkcloud.com:8088

CONNECTED(00000005)
depth=1 C = US, ST = CA, L = San Francisco, O = Splunk, CN = SplunkCommonCA, emailAddress = support@splunk.com
verify error:num=19:self-signed certificate in certificate chain
verify return:1
depth=1 C = US, ST = CA, L = San Francisco, O = Splunk, CN = SplunkCommonCA, emailAddress = support@splunk.com
verify return:1
depth=0 CN = SplunkServerDefaultCert, O = SplunkUser
verify return:1
---
Certificate chain
 0 s:CN = SplunkServerDefaultCert, O = SplunkUser
   i:C = US, ST = CA, L = San Francisco, O = Splunk, CN = SplunkCommonCA, emailAddress = support@splunk.com
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: May 28 17:34:47 2024 GMT; NotAfter: May 28 17:34:47 2027 GMT

We confirmed that for the paid version using the port 443, Splunk is using a valid CA certificate:

echo -n | openssl s_client -connect prd-p-e7xnh.splunkcloud.com:443 | openssl x509 -text -noout
Warning: Reading certificate from stdin since no -in or -new option is given
depth=2 C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Global Root G2
verify return:1
depth=1 C=US, O=DigiCert Inc, CN=DigiCert Global G2 TLS RSA SHA256 2020 CA1
verify return:1
depth=0 C=US, ST=California, L=San Francisco, O=Splunk Inc., CN=*.prd-p-e7xnh.splunkcloud.com
verify return:1
DONE
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            02:ac:04:07:e1:b9:47:0f:a1:83:02:a7:45:99:a4:5f
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=DigiCert Inc, CN=DigiCert Global G2 TLS RSA SHA256 2020 CA1
        Validity
            Not Before: May 28 00:00:00 2024 GMT
            Not After : May 27 23:59:59 2025 GMT
        Subject: C=US, ST=California, L=San Francisco, O=Splunk Inc., CN=*.prd-p-e7xnh.splunkcloud.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Authority Key Identifier:
                74:85:80:C0:66:C7:DF:37:DE:CF:BD:29:37:AA:03:1D:BE:ED:CD:17
            X509v3 Subject Key Identifier:
                35:18:36:ED:18:F5:18:A6:89:90:28:E0:12:AB:14:47:18:37:61:F9
            X509v3 Subject Alternative Name:
                DNS:*.prd-p-e7xnh.splunkcloud.com, DNS:prd-p-e7xnh.splunkcloud.com, DNS:http-inputs-prd-p-e7xnh.splunkcloud.com, DNS:*.http-inputs-prd-p-e7xnh.splunkcloud.com, DNS:akamai-inputs-prd-p-e7xnh.splunkcloud.com, DNS:*.akamai-inputs-prd-p-e7xnh.splunkcloud.com, DNS:http-inputs-ack-prd-p-e7xnh.splunkcloud.com, DNS:*.http-inputs-ack-prd-p-e7xnh.splunkcloud.com, DNS:http-inputs-firehose-prd-p-e7xnh.splunkcloud.com, DNS:*.http-inputs-firehose-prd-p-e7xnh.splunkcloud.com, DNS:*.pvt.prd-p-e7xnh.splunkcloud.com, DNS:pvt.prd-p-e7xnh.splunkcloud.com

Could you use the same certificate for both Trial and Paid version? Why are you using a different one?

Could you please help us. It is blocking us when using Trial accounts. 

Thank you in advance.

[dmitch]

Hi Antonio, to avoid this error (assuming this is a non-production environment) you can set splunkPlatform.insecureSkipVerify to "true" in the values.yaml file you use to deploy the collector: 

https://github.com/signalfx/splunk-otel-collector-chart/blob/320b40a492bc479b12beb4aad20a85e1a9fd12c...

[AntonioJimenez]

Hi @dmitch ,

Thank you for answering. 

I had already tested that in Staging and it works. However, we need the integration with Splunk Cloud Platform in PROD, so we cannot skip TLS verification as it could be a security risk.

Is it possible to fix this issue on Splunk side? Sign the Trial version "prd-p-e7xnh.splunkcloud.com:8088" with the same certificate that the Paid version "prd-p-e7xnh.splunkcloud.com:443". 

We would really appreciate this fix from Splunk.

The rest of observability backend that we have tested have public CA certificate in the target endpoint for Trial Account. 

Thank you in advance. 
Antonio

[phoeneous]

I'm having the same exact issue as @AntonioJimenez and it is also a blocker for us.  Perhaps the author for this article might be able to help?