TLS on HEC ends with RST every time.

[PickleRick]

Hello there.

While troubleshooting a completely other issue I noticed that if I try to send data to HEC input, every connection ends with RST. So the connection looks like this:

1) Normal TCP handshake

2) TLS negotiation

3) Huge chunk of encrypted data from the client to the HEC input (surely, the HTTP request)

4) Small chunk of encrypted data from splunk to the client (apparently - the HTTP response).

5) And here is where it goes weird - RST from the client to the splunk server.

Firstly I suspected the client but tests showed that it's not specific to any particular client. I used rsyslog with omhttp, I used curl - same result.

It seems to be connected to using TLS over the connection because curl-ing over non-TLS http to HEC inputs ends properly with FIN/ACK.

On the other hand however, if I open the connection with openssl s_client and try to perform the request manually, the connection stays open until I close it manually (then s_client sends proper FIN).

So I'm a bit lost. The RST's are most peculiar because you'd either expect a keep-alive (ok, with single request from curl you don't need keep-alive) and more requests over the same connection or a proper FIN/ACK.

It happens regardless of whether it's just server-supplied certificate or a mutual auth setup.

I checked in 8.1.2 at home as well as in 7.3.8 at customer's site - same result.

So I'm a bit puzzled here - is it some bug in HTTPS implementation that the session gets somehow invalidated so that the client resets it or does it have something to do with the configuration? Anyone encountered similar problem?

[PickleRick · Accepted solution]

Interesting.

More digging into that shows that it might not be connected with splunk as such but rather with OpenSSL as such or its use in libcurl as plain curl does the same thing even while connecting to a simple web server, not necessarily to splunk whereas wget closes connection properly.

[PickleRick]

Oh, and there's nothing whatsoever in logs on either side of the connection.

So in general it wouldn't be so much of a concern but if you have many events to send and every single HTTP transaction must involve a completely new TLS negotiation, it's a bit of a pointless work.