Why does using the HTTP Event Collector from C# WCF service fail with SSL error "Could not create SSL/TLS secure channel"?

[michmike]

hi there,

I am using the Splunk Light Cloud service under trial mode to test and make sure i can do what i want with Splunk before purchasing.

if i create a standalone C# program to pump events to Splunk using the HTTP event collector (HEC), it all works fine when i provide the code in c# to ignore certs.

ServicePointManager.ServerCertificateValidationCallback = new
                RemoteCertificateValidationCallback(
                       delegate { return true; }
                    );

However, when I use the exact same code from a WCF service, it always fails with error

The request was aborted: Could not
create SSL/TLS secure channel..

Looking at the network trace from System.Net, the issue is with the certificate of the https endpoint for HEC

System.Net Information: 0 : [3628] SecureChannel#3314304::.ctor(hostname=input-prd-p-kqcmtgqgwbzx.cloud.splunk.com, #clientCertificates=0, encryptionPolicy=RequireEncryption)
System.Net Information: 0 : [3628] Enumerating security packages:
System.Net Information: 0 : [3628]     Negotiate
System.Net Information: 0 : [3628]     NegoExtender
System.Net Information: 0 : [3628]     Kerberos
System.Net Information: 0 : [3628]     NTLM
System.Net Information: 0 : [3628]     TSSSP
System.Net Information: 0 : [3628]     pku2u
System.Net Information: 0 : [3628]     WDigest
System.Net Information: 0 : [3628]     Schannel
System.Net Information: 0 : [3628]     Microsoft Unified Security Protocol Provider
System.Net Information: 0 : [3628]     CREDSSP
System.Net Information: 0 : [3628] SecureChannel#3314304 - Left with 0 client certificates to choose from.
System.Net Information: 0 : [3628] AcquireCredentialsHandle(package = Microsoft Unified Security Protocol Provider, intent  = Outbound, scc     = System.Net.SecureCredential)
System.Net Information: 0 : [3628] InitializeSecurityContext(credential = System.Net.SafeFreeCredential_SECURITY, context = (null), targetName = input-prd-p-kqcmtgqgwbzx.cloud.splunk.com, inFlags = ReplayDetect, SequenceDetect, Confidentiality, AllocateMemory, InitManualCredValidation)
System.Net Information: 0 : [3628] InitializeSecurityContext(In-Buffer length=0, Out-Buffer length=151, returned code=ContinueNeeded).
System.Net.Sockets Verbose: 0 : [3628] Socket#16582792::Send()
System.Net.Sockets Verbose: 0 : [3628] Data from Socket#16582792::Send
System.Net.Sockets Verbose: 0 : [3628] 00000000 : 16 03 03 00 92 01 00 00-8E 03 03 58 9B 42 52 19 : ...........X.BR.
System.Net.Sockets Verbose: 0 : [3628] 00000010 : DD 35 0F 55 FE 35 FE 93-8C 57 F5 EB 90 34 78 FF : .5.U.5...W...4x.
System.Net.Sockets Verbose: 0 : [3628] 00000020 : 33 96 3E 89 8C 6F 70 22-76 92 4B 00 00 0E 00 2F : 3.>..op"v.K..../
System.Net.Sockets Verbose: 0 : [3628] 00000030 : 00 35 00 0A 00 3C 00 3D-00 9C 00 9D 01 00 00 57 : .5...<.=.......W
System.Net.Sockets Verbose: 0 : [3628] 00000040 : 00 00 00 2E 00 2C 00 00-29 69 6E 70 75 74 2D 70 : .....,..)input-p
System.Net.Sockets Verbose: 0 : [3628] 00000050 : 72 64 2D 70 2D 6B 71 63-6D 74 67 71 67 77 62 7A : rd-p-kqcmtgqgwbz
System.Net.Sockets Verbose: 0 : [3628] 00000060 : 78 2E 63 6C 6F 75 64 2E-73 70 6C 75 6E 6B 2E 63 : x.cloud.splunk.c
System.Net.Sockets Verbose: 0 : [3628] 00000070 : 6F 6D 00 0D 00 14 00 12-06 01 06 03 04 01 05 01 : om..............
System.Net.Sockets Verbose: 0 : [3628] 00000080 : 02 01 04 03 05 03 02 03-02 02 00 23 00 00 00 17 : ...........#....
System.Net.Sockets Verbose: 0 : [3628] 00000090 : 00 00 FF 01 00 01 00                            : .......
System.Net.Sockets Verbose: 0 : [3628] Exiting Socket#16582792::Send()  -> Int32#151
System.Net.Sockets Verbose: 0 : [3628] Socket#16582792::Receive()
System.Net.Sockets Verbose: 0 : [3628] Data from Socket#16582792::Receive
System.Net.Sockets Verbose: 0 : [3628] 00000000 : 15 03 03 00 02                                  : .....
System.Net.Sockets Verbose: 0 : [3628] Exiting Socket#16582792::Receive()   -> Int32#5
System.Net.Sockets Verbose: 0 : [3628] Socket#16582792::Receive()
System.Net.Sockets Verbose: 0 : [3628] Data from Socket#16582792::Receive
System.Net.Sockets Verbose: 0 : [3628] 00000005 : 02 28                                           : .(
System.Net.Sockets Verbose: 0 : [3628] Exiting Socket#16582792::Receive()   -> Int32#2
System.Net Information: 0 : [3628] InitializeSecurityContext(credential = System.Net.SafeFreeCredential_SECURITY, context = 1a9d4c90:542a58ac90, targetName = input-prd-p-kqcmtgqgwbzx.cloud.splunk.com, inFlags = ReplayDetect, SequenceDetect, Confidentiality, AllocateMemory, InitManualCredValidation)
System.Net Information: 0 : [3628] InitializeSecurityContext(In-Buffers count=2, Out-Buffer length=0, returned code=IllegalMessage).
System.Net.Sockets Verbose: 0 : [3628] Socket#16582792::Dispose()
System.Net Error: 0 : [3628] Exception in HttpWebRequest#49840803:: - The request was aborted: Could not create SSL/TLS secure channel..
System.Net Error: 0 : [3628] Exception in HttpWebRequest#49840803::EndGetRequestStream - The request was aborted: Could not create SSL/TLS secure channel..
System.Net.Sockets Verbose: 0 : [9168] Socket#33711845::Send()

any ideas on how to get around this?
thanks in advance!

[michmike · Accepted solution]

i was finally able to solve this and updating this page for future folks.
In trying to understand the issue, i concentrated in 3 areas that could possibly cause the response of an IllegalMessage (Win32 SChannel Native Error Code: 0x80090326)

1. Did the site use ECC (elliptic curves cryptography) in their certificate and my server could not support it
2. Was SSL renegotiation not working
3. Did the ciphers my computer was producing not matching the certificate expectations

wireshark and microsoft message analyzer helped a little bit, but since I did not have the private key of the Splunk certificate, i could not decrypt a lot of the content. Fiddler, with the man in the middle proxy, gave me some useful data as well.

Eventually, #1 was not an issue. #2 was an issue and I discovered https://support.microsoft.com/en-us/help/2745582/.net-framework-4.5-hotfix-resolves-httpwebrequest-a.... With that in mind, I updated my computers to .NET 4.6.2

Once I got past all the other issues, #3 became my problem. According to this article, https://technet.microsoft.com/en-us/library/dn786419(v=ws.11).aspx, Windows Server 2012 R2 and Win2k16 support the same TLS and cipher suits. After lots of debugging, i discovered our domain policy had a group policy that restricted the ciphers available for use and demanded a specific order depending on the Operating System you were using. After manual trial and error, I was able to pinpoint the exact 3 ciphers needed by the Splunk Light public endpoint for HTTP event collection:

TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P256
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P384

the fact that was i executing the code from a WCF service ended up playing no role in this.

[michmike]

any update here? i have figured that the code works great from windows 10 machines (windows server 2016), but not from earlier operating systems. a trace using fiddler is showing the following details

Request

CONNECT input-prd-p-kqcmtgqgwbzx.cloud.splunk.com:8088 HTTP/1.1
Host: input-prd-p-kqcmtgqgwbzx.cloud.splunk.com:8088

A SSLv3-compatible ClientHello handshake was found. Fiddler extracted the parameters below.

Version: 3.3 (TLS/1.2)
Random: 58 9C 87 02 8B C4 C8 CF 51 E6 39 D6 75 81 FA 39 EF 87 7B 30 44 ED 45 D6 06 33 FD 68 16 0A F8 0E
"Time": 5/7/1971 1:23:36 AM
SessionID: empty
Extensions: 
    server_name input-prd-p-kqcmtgqgwbzx.cloud.splunk.com
    signature_algs  sha512_rsa, sha512_ecdsa, sha256_rsa, sha384_rsa, sha1_rsa, sha256_ecdsa, sha384_ecdsa, sha1_ecdsa, sha1_dsa
    SessionTicket   empty
    extended_master_secret  empty
    renegotiation_info  00
Ciphers: 
    [002F]  TLS_RSA_AES_128_SHA
    [0035]  TLS_RSA_AES_256_SHA
    [000A]  SSL_RSA_WITH_3DES_EDE_SHA
    [003C]  TLS_RSA_WITH_AES_128_CBC_SHA256
    [003D]  TLS_RSA_WITH_AES_256_CBC_SHA256
    [009C]  TLS_RSA_WITH_AES_128_GCM_SHA256
    [009D]  TLS_RSA_WITH_AES_256_GCM_SHA384

Compression: 
    [00]    NO_COMPRESSION

Response

HTTP/1.1 200 Connection Established
FiddlerGateway: Direct
StartTime: 20:04:28.030
Connection: close

fiddler.network.https> HTTPS handshake to input-prd-p-kqcmtgqgwbzx.cloud.splunk.com (for #2) failed. System.Security.Authentication.AuthenticationException A call to SSPI failed, see inner exception. < The message received was unexpected or badly formatted

Win32 (SChannel) Native Error Code: 0x80090326

[michmike]

A successful run gives me this output in fiddler

CONNECT input-prd-p-kqcmtgqgwbzx.cloud.splunk.com:8088 HTTP/1.1
Host: input-prd-p-kqcmtgqgwbzx.cloud.splunk.com:8088
Connection: Keep-Alive

A SSLv3-compatible ClientHello handshake was found. Fiddler extracted the parameters below.

Version: 3.3 (TLS/1.2)
Random: 58 9C 87 94 28 C9 40 29 54 CE C4 98 56 4A 09 D2 CC F9 99 FC 77 F0 38 C9 C7 80 66 02 CD 44 1E E6
"Time": 12/18/2048 9:42:32 AM
SessionID: empty
Extensions: 
    server_name input-prd-p-kqcmtgqgwbzx.cloud.splunk.com
    elliptic_curves unknown [0x1D), secp256r1 [0x17], secp384r1 [0x18]
    ec_point_formats    uncompressed [0x0]
    signature_algs  sha256_rsa, sha384_rsa, sha1_rsa, sha256_ecdsa, sha384_ecdsa, sha1_ecdsa, sha1_dsa, sha512_rsa, sha512_ecdsa
    SessionTicket   empty
    extended_master_secret  empty
    renegotiation_info  00
Ciphers: 
    [C02C]  TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
    [C02B]  TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
    [C030]  TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
    [C02F]  TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
    [009F]  TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
    [009E]  TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
    [C024]  TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
    [C023]  TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
    [C028]  TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
    [C027]  TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
    [C00A]  TLS1_CK_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
    [C009]  TLS1_CK_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
    [C014]  TLS1_CK_ECDHE_RSA_WITH_AES_256_CBC_SHA
    [C013]  TLS1_CK_ECDHE_RSA_WITH_AES_128_CBC_SHA
    [0039]  TLS_DHE_RSA_WITH_AES_256_SHA
    [0033]  TLS_DHE_RSA_WITH_AES_128_SHA
    [009D]  TLS_RSA_WITH_AES_256_GCM_SHA384
    [009C]  TLS_RSA_WITH_AES_128_GCM_SHA256
    [003D]  TLS_RSA_WITH_AES_256_CBC_SHA256
    [003C]  TLS_RSA_WITH_AES_128_CBC_SHA256
    [0035]  TLS_RSA_AES_256_SHA
    [002F]  TLS_RSA_AES_128_SHA
    [000A]  SSL_RSA_WITH_3DES_EDE_SHA
    [006A]  TLS_DHE_DSS_WITH_AES_256_CBC_SHA256
    [0040]  TLS_DHE_DSS_WITH_AES_128_CBC_SHA256
    [0038]  TLS_DHE_DSS_WITH_AES_256_SHA
    [0032]  TLS_DHE_DSS_WITH_AES_128_SHA
    [0013]  SSL_DHE_DSS_WITH_3DES_EDE_SHA

Compression: 
    [00]    NO_COMPRESSION

[starcher]

Splunk by default uses self signed certs. That is why you have to not validate to work. You would have to use the ca certificate chain for that default cert, replace with your own certs or leave validation off. I don't know if the light cloud uses same default certs as a stand alone. You can install local and look in $SPLUNK_HOME$/etc/auth

[michmike]

hi there starcher, you are 100% correct on the self-signed certs. that's why i skip cert validation using this code.

ServicePointManager.ServerCertificateValidationCallback = new
             RemoteCertificateValidationCallback(
                    delegate { return true; }
                 );

However, i am still failing to get this to work from a WCF service

[starcher]

Sorry I know nothing of that service or C#. I pinged the HEC team that this thread exists. Not sure if one of them can look and have any ideas.

[FastTrack]

@starcherI know this is an older thread, but it seems that this is still very much an issue. We just signed up to Splunk to test the HEC with our webhooks. However all calls fail with the "The underlying connection was closed" message. I've tracked it down to the exact same point as your post here: Splunk uses self-signed certificates.

It's not really a viable option to disable certificate validation in our production environment. Do you know of any other way to get around this? Or perhaps any reply from the Splunk HEC team?

Thanks a lot.

[starcher]

Never heard more. Try not running default certs on a hec receiver. Or ask in community slack hec channel. I’m unfamiliar with c# implementation for hec. 

[michmike]

thanks for the assistance

[michmike]

I guess the key line in the log is the following

InitializeSecurityContext(In-Buffers count=2, Out-Buffer length=0, returned code=IllegalMessage).

In a successful log, it says

System.Net Information: 0 : [17520] InitializeSecurityContext(In-Buffers count=2, Out-Buffer length=0, returned code=ContinueNeeded).

I already set this flag in my code:

ServicePointManager.Expect100Continue = true;