[HEC] Endpoint : TCP reset session all time

[shinigami35]

Hello Everyone,

I need your help about a problem with Splunk HEC. I use the endpoint "event" to send logs into multiple HEC.

I use Rsyslog (Omhttp) to send those logs.

But I have a problem, each time I send logs to the HEC, this last one sent back a "FIN TCP" to my source server and close the TCP session . I set up my source server to send Keep-Alive TCP session between it and the HEC, but this parameter does not work. At each request, the HEC close the session and ignore the keep-alive.

I try to apply this workaround "Solved: Splunk HEC closes connection instead of re-using i... - Splunk Community" but this is the same behaviour ...

I need to understand why the HEC close TCP session all time, because I need to send my logs in HTTPS and with this behaviour, I have a TLS handshake at each log sent.

Best regards,

Shini

[PickleRick]

So this is not RST. I think I saw rsyslog as client sending RSTs but that might have been over RELP then. What is your forceHttp10 value?

forceHttp10 = [auto|never|always]
* Whether or not the REST HTTP server forces clients that connect
  to it to use the HTTP 1.0 specification for web communications.
* When set to "always", the REST HTTP server does not use some
  HTTP 1.1 features such as persistent connections or chunked
  transfer encoding.
* When set to "auto", it does this only if the client did not send
  a User-Agent header, or if the user agent is known to have bugs
  in its support of HTTP/1.1.
* When set to "never" it always allows HTTP 1.1, even to
  clients it suspects might be buggy.
* Default: auto

[shinigami35]

Hello @PickleRick,

The value is set to auto. But I do not think this is the main problem.

For me, the HEC has a "hidden" params to force the HEC accepting Keep-Alive because as show in the previous comment, the HEC answers with "Connection: close" at each request.

Best regards,

Shini

[PickleRick]

As I read the description to that option - if set to auto Splunk will enforce http/1.0 (possibly adding the Connection: close header for good measure or simply because it's easier to program it this way if there is no User-Agent header in the request. And as you've shown, there was no such header. Try setting the header to something. You're probably using the httpheaderkey and httpheadervalue parameters for passing the token so just use httpheaders option.

I must test this myself as well. Might help in my future installations.

[shinigami35]

Hello,

I checked if my server used HTTP/1.0 version, and not the case. The connection between my server and the HEC is in HTTP/1.1.

Also, the HTTP header "Connection: Keep-Alive" is correctly added in the HTTP header. I opened a case to the Splunk support to ask the why the HEC ignore keep-alive.

Best regards

Shini

[PickleRick]

What is your rsyslog omhttp action config?

In my tests I don't see rsyslog sending "Connecton: Keep-alive" on its own.  If you add it as custom header it doesn't mean rsyslog will know how to handle the persistent connection.

[shinigami35]

Hello,

The keep-alive params in omhttp is in its own code. Those default params are :

• CURLOPT_TCP_KEEPALIVE -> 1 (activation of TCP keep-alive)
• CURLOPT_TCP_KEEPIDLE -> 120s 
• CURLOPT_TCP_KEEPINTVL -> 60s

Also the header "Connectio: Keep-alive" is normaly not necesary if you use HTTP/1.1, this header is for HTTP/1.0. I tested with the addition of this  HTTP header to my request to check if this parms change something but it was an failed.

Also, I do not know why the POST answer of the HEC server code 200 add every time the header "Connection: close". It seems like the implementing of the HEC server is a mix between HTTP/1.1 and HTTP.1.0.

Best regards,

Shini 

[PickleRick]

OK. It seems the docs for http input are correct. If you do not include User-Agent header (or have a header which is on some sort of blacklist), the input works as HTTP/1.0 input. If you do insert a User-Agent header, it works with persistent connection.

If I do

action(
name="splunk-hec"
template="splunk-hec-nested"
type="omhttp"
Server="172.16.0.3"
Serverport="8088"
useHttps="off"
healthchecktimeout="1000"
httpheaderkey="Authorization"
httpheadervalue="Splunk 12345678-1234-1234-1234-123412341234"
restpath="services/collector"
checkpath="services/collector/health"
batch="on"
batch.format="newline"
batch.maxsize="256"
retry="off"
reloadonhup="on"
)

The HEC input closes server after each batch from rsyslog.

If I add the httpheaders parameter making it

action(
name="splunk-hec"
template="splunk-hec-nested"
type="omhttp"
Server="172.16.0.3"
Serverport="8088"
useHttps="off"
healthchecktimeout="1000"
httpheaderkey="Authorization"
httpheadervalue="Splunk 12345678-1234-1234-1234-123412341234"
httpheaders=["User-Agent: Rsyslog"]
restpath="services/collector"
checkpath="services/collector/health"
batch="on"
batch.format="newline"
batch.maxsize="256"
retry="off"
reloadonhup="on"
)

the connection is kept open and reused over separate http requests.

[shinigami35]

Hello,

I did the same test as you and I have got the same result.

When you use the HEC you need to specify an "User-Agent" to use the HTTP/1.1 version.

I did this test :

action(name="omhttptest"
type="omhttp"
server="10.10.10.10"
serverport="8088"
profile="hec:splunk:event"
template="tpl_omhttp_json_hec"
token="X-X-X-X"
errorfile="/var/log/omhttp_test_errors.log"
statsbysenders="on"
batch="on"
httpheaders=["User-Agent: Rsyslog"]
)

This behaviour is also available when you force HEC to accept only HTTP/1.1 with the params

forceHttp10 = never

For me, this is not a good implementation of the HTTP/1.1 protocol because, in this version of HTTP the params "User-Agent" is "should" and not "must"

For next step, I will ask Splunk why they decided to do like that.

I will also do another PR to Rsyslog to patch this behaviour and add a custom User-Agent into the code of Omhttp in the profile HEC Splunk.

Thanks for your help @PickleRick 

Best regards,

[PickleRick]

To be honest, HEC is not supposed to be a full-fledged HTTP server and therefore it might not implement a very big set of features. Just what is needed for the input to work decently enough. 🙂

It does seem to work as described in the docs though and that's - again - good enough. (although an explicit list of "unworthy" clients would be useful).

[PickleRick]

I think I might have seen this behaviour. Unfortunately if it's the client just sending RST (RST is not the same as FIN), no amount of server-side settings will help. You might want to:

1. Try to set up some dummy HTTP receiver to verify if omhttp does it with it as well.

2. Ask on rsyslog mailing list.

As a partial walkaround you might want to use batch mode with the omhttp action so rsyslog sends bigger batches of data and doesn't waste as much bandwidth for http headers, tls negotiation and tcp handshake. omhttp can handle batches of decent sizes (like 256 or more) quite well. 

[shinigami35]

Hello @PickleRick,

I already check the omhttp behaviour during a TCP session.

The flow is like that :

Rsyslog send SYN -> HEC send SYN ACK -> Rsyslog send PUSH (batch data of 1 Mb) -> HEC send PUSH (with answer {"text":"Success","code":0}) -> Rsyslog receive data -> HEC send TCP FIN -> Rsyslog answer TCP FIN 

Example of Header send by Rsyslog to the POST Request : (Connection: Keep-Alive)

POST /services/collector/event HTTP/1.1
Host: X.X.X.X:8088
Accept: */*
Content-Type: text/plain
Connection: Keep-Alive
Authorization: Splunk XXX-XXX-XXX-XXX-XXXX
Content-Length: 624

The HEC always anwer like that :

Date: Thu, 18 Dec 2025 20:40:34 GMT
Content-Type: application/json; charset=UTF-8
X-Content-Type-Options: nosniff
Content-Length: 27
Vary: Authorization
Connection: Close
X-Frame-Options: SAMEORIGIN
Server: Splunkd

So this is little bit "Magic".

And if use HTTPS, this is an overload on my server, because I have one TLS handshake by each HTTP request...

I do not know if someone has this "problem" 

Best regards,

Shini

[livehybrid]

Hi @shinigami35 

If there is an Firewall / F5, Nginx, or HAProxy between Rsyslog and Splunk, the proxy often terminates the TCP session.

Check if the proxy is injecting a Connection: close header and ensure the proxy's "Idle Timeout" is longer than the interval between your Rsyslog batches. Also check the proxy supports HTTP/1.1 (HTTP/1.0 does not support persistent connections by default).

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[shinigami35]

Hello @livehybrid ,

In this case , there is not Proxy or Firewall between my server and the HEC server (hosted on an indexer).

I set up the idle as good as it needs to be and the KeepAlive (Rsyslog side) is set up at 120 seconds.

In my case I do more than one thousand HTTP request every seconds so I really need the TCP session to stay opened.

Headers are correctly set and I use HTTP 1.1 version. So I do not know if I need to set the params « forceHttp10 » to « never » or another param.

best regards

shini