Splunk HEC closes connection instead of re-using it

[onlineops]

Our apps send data to the Splunk HEC via HTTP POSTS. The apps are configured to use a connection pool, but after sending data to Splunk (via HTTP POSTS), the Splunk server responds with a Status 200 and the "Connection: Close" header. This instructs our apps to close their connection instead of reusing the connection.

How can I stop this behavior? Right now it's constantly re-creating a connection thousands of times instead of just re-using the same connection.

[onlineops · Accepted solution]

To fix this issue, we had our client insert the "Connection: Keep-Alive" header into the HTTP POST requests. This instructed the Splunk server to keep the connection alive.

[PickleRick]

Interesting find. It's inconsistent with the docs so it calls for a support case or at least a docs feedback.

[PickleRick]

Are your clients sending proper HTTP/1.1. Splunk should support keep-alive out of the box.

[onlineops]

Thank you for replying. Yes, the client is using HTTP 1.1 when sending the HTTP POSTS. This was verified within the packet capture.

[PickleRick]

Well, this says that Splunk should normally behave properly with HTTP/1.1

https://docs.splunk.com/Documentation/Splunk/latest/Data/TroubleshootHTTPEventCollector#Detect_scali...

Another thing to consider.

forceHttp10 = [auto|never|always]
* Whether or not the REST HTTP server forces clients that connect
  to it to use the HTTP 1.0 specification for web communications.
* When set to "always", the REST HTTP server does not use some
  HTTP 1.1 features such as persistent connections or chunked
  transfer encoding.
* When set to "auto", it does this only if the client did not send
  a User-Agent header, or if the user agent is known to have bugs
  in its support of HTTP/1.1.
* When set to "never" it always allows HTTP 1.1, even to
  clients it suspects might be buggy.
* Default: auto