Turning off chunked encoding in Universal Forwarder

[jni]

Hi,

We're setting up a Splunk enterprise instance in an air-gapped environment. 
In addition to this, the server is situated behind a diode, making all traffic one-way.

I've gotten the Splunk Universal Forwarders to send logs over http to the HEC, but the diode doesn't support chunked http-encoding. It isn't possible to turn off http 1.1-support in the diode. 

In the server, there's the option "forceHttp10", but since the client and server doesn't negotiate the http-version it has no effect.

Is there an option in the UF to turn off http 1.1 or chunking for httpout?

TIA

Johan

[PickleRick · Accepted solution]

The data sent by httpout is _not_ your normal HEC. True, it uses the same port and the same tokens but the transmission method is different. It's actually more of a s2s protocol embedded in HTTP requests. Therefore I wouldn't be very optimistic about "downgrading" HTTP version/features on the fly.

[jni]

Thanks. 

Could you elaborate?

My understanding is that  [httpout] will tunnel s2s over http to a HEC on the server.

This give us one-way communication possible, since the diode accepts the http-session and closes it with a "200 ok"-reply.

[PickleRick]

I mean that normally HEC input expects separate events in one of supported formats and you can see (as long as you decode the transport layer) the payload in clear text. httpout indeed does some magic to send s2s over HTTP but the contents are not readable in plain and cannot easily be manipulated. S2S can send both cooked and parsed data and as far as I remember also supports some in-transit compression. It also supports acknowledging within this protocol (not on a separate endpoint as with HEC).

[jni]

Thanks for the explanation! 

Sending s2s over http works fine for us, and especially since log data can't be manipulated in-flight. 

[PickleRick]

Yes. s2s over http works relatively OK in a standard environment and over proxies/LB and such (that's why it was introduced I think - it's way easier for customers to allow outgoing http traffic to Cloud than to open ports to some unknown protocol - raises much less questions). But there is no guarantee that it will work when you try to manipulate the payload.

[dural_yyz]

I can not find anything in the outputs.conf that will allow you to control the HTTP version sourced at the UF itself.  Splunk documentation implies a LB can/should be used and can control HTTP version.  Their example is NGINX but there are others out there which may or may not support in the same fashion.

https://docs.splunk.com/Documentation/Forwarder/9.4.0/Forwarder/Configureforwardingwithoutputs.conf#...

[jni]

Thanks, that's what I've found as well. 

I did tunnel the data through a nginx reverse-proxy, and that forwarded the data as "complete" and not "chunked". 

The problem is that this will change the design of the network, and will require a new approval. So any workaround that don't require design-changes would be great.

//Johan