topic Splunk HEC closes connection instead of re-using it in Getting Data In

Splunk HEC closes connection instead of re-using it

onlineops — Fri, 08 Nov 2024 19:19:00 GMT

Our apps send data to the Splunk HEC via HTTP POSTS. The apps are configured to use a connection pool, but after sending data to Splunk (via HTTP POSTS), the Splunk server responds with a Status 200 and the "Connection: Close" header. This instructs our apps to close their connection instead of reusing the connection.

How can I stop this behavior? Right now it's constantly re-creating a connection thousands of times instead of just re-using the same connection.

Re: Splunk HEC closes connection instead of re-using it

PickleRick — Fri, 08 Nov 2024 19:48:31 GMT

Are your clients sending proper HTTP/1.1. Splunk should support keep-alive out of the box.

Re: Splunk HEC closes connection instead of re-using it

onlineops — Fri, 08 Nov 2024 20:10:44 GMT

Thank you for replying. Yes, the client is using HTTP 1.1 when sending the HTTP POSTS. This was verified within the packet capture.

Re: Splunk HEC closes connection instead of re-using it

PickleRick — Fri, 08 Nov 2024 21:02:49 GMT

Well, this says that Splunk should normally behave properly with HTTP/1.1

https://docs.splunk.com/Documentation/Splunk/latest/Data/TroubleshootHTTPEventCollector#Detect_scaling_problems

Another thing to consider.

forceHttp10 = [auto|never|always]
* Whether or not the REST HTTP server forces clients that connect
  to it to use the HTTP 1.0 specification for web communications.
* When set to "always", the REST HTTP server does not use some
  HTTP 1.1 features such as persistent connections or chunked
  transfer encoding.
* When set to "auto", it does this only if the client did not send
  a User-Agent header, or if the user agent is known to have bugs
  in its support of HTTP/1.1.
* When set to "never" it always allows HTTP 1.1, even to
  clients it suspects might be buggy.
* Default: auto

Re: Splunk HEC closes connection instead of re-using it

onlineops — Fri, 08 Nov 2024 22:37:12 GMT

To fix this issue, we had our client insert the "Connection: Keep-Alive" header into the HTTP POST requests. This instructed the Splunk server to keep the connection alive.

Re: Splunk HEC closes connection instead of re-using it

PickleRick — Sat, 09 Nov 2024 07:19:45 GMT

Interesting find. It's inconsistent with the docs so it calls for a support case or at least a docs feedback.