topic Re: [HEC] Endpoint : TCP reset session all time in Getting Data In

[HEC] Endpoint : TCP reset session all time

shinigami35 — Thu, 18 Dec 2025 17:01:12 GMT

Hello Everyone,

I need your help about a problem with Splunk HEC. I use the endpoint "event" to send logs into multiple HEC.

I use Rsyslog (Omhttp) to send those logs.

But I have a problem, each time I send logs to the HEC, this last one sent back a "FIN TCP" to my source server and close the TCP session . I set up my source server to send Keep-Alive TCP session between it and the HEC, but this parameter does not work. At each request, the HEC close the session and ignore the keep-alive.

I try to apply this workaround "Solved: Splunk HEC closes connection instead of re-using i... - Splunk Community" but this is the same behaviour ...

I need to understand why the HEC close TCP session all time, because I need to send my logs in HTTPS and with this behaviour, I have a TLS handshake at each log sent.

Best regards,

Shini

Re: [HEC] Endpoint : TCP reset session all time

livehybrid — Thu, 18 Dec 2025 18:32:43 GMT

Hi @shinigami35

If there is an Firewall / F5, Nginx, or HAProxy between Rsyslog and Splunk, the proxy often terminates the TCP session.

Check if the proxy is injecting a Connection: close header and ensure the proxy's "Idle Timeout" is longer than the interval between your Rsyslog batches. Also check the proxy supports HTTP/1.1 (HTTP/1.0 does not support persistent connections by default).

🌟 Did this answer help you? If so, please consider:

Adding karma to show it was useful
Marking it as the solution if it resolved your issue
Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

Re: [HEC] Endpoint : TCP reset session all time

PickleRick — Thu, 18 Dec 2025 19:56:29 GMT

I think I might have seen this behaviour. Unfortunately if it's the client just sending RST (RST is not the same as FIN), no amount of server-side settings will help. You might want to:

1. Try to set up some dummy HTTP receiver to verify if omhttp does it with it as well.

2. Ask on rsyslog mailing list.

As a partial walkaround you might want to use batch mode with the omhttp action so rsyslog sends bigger batches of data and doesn't waste as much bandwidth for http headers, tls negotiation and tcp handshake. omhttp can handle batches of decent sizes (like 256 or more) quite well.

Re: [HEC] Endpoint : TCP reset session all time

shinigami35 — Thu, 18 Dec 2025 19:58:43 GMT

Hello @livehybrid ,

In this case , there is not Proxy or Firewall between my server and the HEC server (hosted on an indexer).

I set up the idle as good as it needs to be and the KeepAlive (Rsyslog side) is set up at 120 seconds.

In my case I do more than one thousand HTTP request every seconds so I really need the TCP session to stay opened.

Headers are correctly set and I use HTTP 1.1 version. So I do not know if I need to set the params « forceHttp10 » to « never » or another param.

best regards

shini

Re: [HEC] Endpoint : TCP reset session all time

shinigami35 — Thu, 18 Dec 2025 20:48:44 GMT

Hello @PickleRick,

I already check the omhttp behaviour during a TCP session.

The flow is like that :

Rsyslog send SYN -> HEC send SYN ACK -> Rsyslog send PUSH (batch data of 1 Mb) -> HEC send PUSH (with answer {"text":"Success","code":0}) -> Rsyslog receive data -> HEC send TCP FIN -> Rsyslog answer TCP FIN

Example of Header send by Rsyslog to the POST Request : (Connection: Keep-Alive)

POST /services/collector/event HTTP/1.1
Host: X.X.X.X:8088
Accept: */*
Content-Type: text/plain
Connection: Keep-Alive
Authorization: Splunk XXX-XXX-XXX-XXX-XXXX
Content-Length: 624

The HEC always anwer like that :

Date: Thu, 18 Dec 2025 20:40:34 GMT
Content-Type: application/json; charset=UTF-8
X-Content-Type-Options: nosniff
Content-Length: 27
Vary: Authorization
Connection: Close
X-Frame-Options: SAMEORIGIN
Server: Splunkd

So this is little bit "Magic".

And if use HTTPS, this is an overload on my server, because I have one TLS handshake by each HTTP request...

I do not know if someone has this "problem"

Best regards,

Shini

Re: [HEC] Endpoint : TCP reset session all time

PickleRick — Fri, 19 Dec 2025 07:43:22 GMT

So this is not RST. I think I saw rsyslog as client sending RSTs but that might have been over RELP then. What is your forceHttp10 value?

forceHttp10 = [auto|never|always]
* Whether or not the REST HTTP server forces clients that connect
  to it to use the HTTP 1.0 specification for web communications.
* When set to "always", the REST HTTP server does not use some
  HTTP 1.1 features such as persistent connections or chunked
  transfer encoding.
* When set to "auto", it does this only if the client did not send
  a User-Agent header, or if the user agent is known to have bugs
  in its support of HTTP/1.1.
* When set to "never" it always allows HTTP 1.1, even to
  clients it suspects might be buggy.
* Default: auto

Re: [HEC] Endpoint : TCP reset session all time

shinigami35 — Fri, 19 Dec 2025 10:04:05 GMT

Hello @PickleRick,

The value is set to auto. But I do not think this is the main problem.

For me, the HEC has a "hidden" params to force the HEC accepting Keep-Alive because as show in the previous comment, the HEC answers with "Connection: close" at each request.

Best regards,

Shini

Re: [HEC] Endpoint : TCP reset session all time

PickleRick — Fri, 19 Dec 2025 19:11:45 GMT

As I read the description to that option - if set to auto Splunk will enforce http/1.0 (possibly adding the Connection: close header for good measure or simply because it's easier to program it this way if there is no User-Agent header in the request. And as you've shown, there was no such header. Try setting the header to something. You're probably using the httpheaderkey and httpheadervalue parameters for passing the token so just use httpheaders option.

I must test this myself as well. Might help in my future installations.

Re: [HEC] Endpoint : TCP reset session all time

shinigami35 — Mon, 22 Dec 2025 09:01:42 GMT

Hello,

I checked if my server used HTTP/1.0 version, and not the case. The connection between my server and the HEC is in HTTP/1.1.

Also, the HTTP header "Connection: Keep-Alive" is correctly added in the HTTP header. I opened a case to the Splunk support to ask the why the HEC ignore keep-alive.

Best regards

Shini

Re: [HEC] Endpoint : TCP reset session all time

PickleRick — Mon, 22 Dec 2025 11:56:31 GMT

What is your rsyslog omhttp action config?

In my tests I don't see rsyslog sending "Connecton: Keep-alive" on its own. If you add it as custom header it doesn't mean rsyslog will know how to handle the persistent connection.

Re: [HEC] Endpoint : TCP reset session all time

shinigami35 — Mon, 22 Dec 2025 12:59:06 GMT

Hello,

The keep-alive params in omhttp is in its own code. Those default params are :

CURLOPT_TCP_KEEPALIVE -> 1 (activation of TCP keep-alive)
CURLOPT_TCP_KEEPIDLE -> 120s
CURLOPT_TCP_KEEPINTVL -> 60s

Also the header "Connectio: Keep-alive" is normaly not necesary if you use HTTP/1.1, this header is for HTTP/1.0. I tested with the addition of this HTTP header to my request to check if this parms change something but it was an failed.

Also, I do not know why the POST answer of the HEC server code 200 add every time the header "Connection: close". It seems like the implementing of the HEC server is a mix between HTTP/1.1 and HTTP.1.0.

Best regards,

Shini

Re: [HEC] Endpoint : TCP reset session all time

PickleRick — Mon, 22 Dec 2025 15:10:13 GMT

OK. It seems the docs for http input are correct. If you do not include User-Agent header (or have a header which is on some sort of blacklist), the input works as HTTP/1.0 input. If you do insert a User-Agent header, it works with persistent connection.

If I do

action(
name="splunk-hec"
template="splunk-hec-nested"
type="omhttp"
Server="172.16.0.3"
Serverport="8088"
useHttps="off"
healthchecktimeout="1000"
httpheaderkey="Authorization"
httpheadervalue="Splunk 12345678-1234-1234-1234-123412341234"
restpath="services/collector"
checkpath="services/collector/health"
batch="on"
batch.format="newline"
batch.maxsize="256"
retry="off"
reloadonhup="on"
)

The HEC input closes server after each batch from rsyslog.

If I add the httpheaders parameter making it

action(
name="splunk-hec"
template="splunk-hec-nested"
type="omhttp"
Server="172.16.0.3"
Serverport="8088"
useHttps="off"
healthchecktimeout="1000"
httpheaderkey="Authorization"
httpheadervalue="Splunk 12345678-1234-1234-1234-123412341234"
httpheaders=["User-Agent: Rsyslog"]
restpath="services/collector"
checkpath="services/collector/health"
batch="on"
batch.format="newline"
batch.maxsize="256"
retry="off"
reloadonhup="on"
)

the connection is kept open and reused over separate http requests.

Re: [HEC] Endpoint : TCP reset session all time

shinigami35 — Mon, 22 Dec 2025 16:46:08 GMT

Hello,

I did the same test as you and I have got the same result.

When you use the HEC you need to specify an "User-Agent" to use the HTTP/1.1 version.

I did this test :

action(name="omhttptest"
type="omhttp"
server="10.10.10.10"
serverport="8088"
profile="hec:splunk:event"
template="tpl_omhttp_json_hec"
token="X-X-X-X"
errorfile="/var/log/omhttp_test_errors.log"
statsbysenders="on"
batch="on"
httpheaders=["User-Agent: Rsyslog"]
)

This behaviour is also available when you force HEC to accept only HTTP/1.1 with the params

forceHttp10 = never

For me, this is not a good implementation of the HTTP/1.1 protocol because, in this version of HTTP the params "User-Agent" is "should" and not "must"

For next step, I will ask Splunk why they decided to do like that.

I will also do another PR to Rsyslog to patch this behaviour and add a custom User-Agent into the code of Omhttp in the profile HEC Splunk.

Thanks for your help @PickleRick

Best regards,

Re: [HEC] Endpoint : TCP reset session all time

PickleRick — Mon, 22 Dec 2025 19:23:53 GMT

To be honest, HEC is not supposed to be a full-fledged HTTP server and therefore it might not implement a very big set of features. Just what is needed for the input to work decently enough. 🙂

It does seem to work as described in the docs though and that's - again - good enough. (although an explicit list of "unworthy" clients would be useful).

Re: [HEC] Endpoint : TCP reset session all time

shinigami35 — Wed, 24 Dec 2025 15:37:17 GMT

Hello,

Full agree with you about this point.

Also, I write to Splunk support for asking them why they decided to implement the HEC server with a mix of HTTP/1.0 and HTTP/1.1 version without repecting the HTTP RFC. I am waiting the answer !

Also, I modify the omhttp module of Rsyslog to add an User-Agent header into all ost request for Splunk HEC to avoid this behavior for user like us.

Information for you : In the version 8.2512 Rsyslog, I added the support of HEC Splunk server directly into the Omhttp code, so I simplify the use of omhttp and Splunk HEC.

And in the next version, I rewrite the core of omhttp to optimize this module and to be able to send about ten Tb of data to multiple HEC servers. (I already use it on my side and it is faster than an UF).

Best regards,

Shini

Re: [HEC] Endpoint : TCP reset session all time

shinigami35 — Tue, 13 Jan 2026 14:29:10 GMT

Hello,

This is the answer of Splunk Support :

This behaviour is as designed. 
HTTP requests that do not supply User-Agent header are not trusted to be "true" HTTP/1.1 peers,
 therefore they are closed on completion. To take advantage of persistent connections, 
please set the User-Agent header in the request.

Best regards,

Shini

Re: [HEC] Endpoint : TCP reset session all time

PickleRick — Tue, 13 Jan 2026 14:40:02 GMT

Well... to some extent it makes sense.

RFC says that HTTP/1.1 client SHOULD include User-Agent header. So if it doesn't, it's either not a fully implemented HTTP/1.1 client or might have other features also not fully implemented (persistent connections are also a SHOULD requirement for HTTP/1.1 implementation).