Logs compression in Splunk HEC spring boot

[priyanka2887]

Hi Team,

Can we compress the logs using Splunk HEC HttpEventCollectorLogbackAppender? Please guide here, how to compress the logs using splunk hec configuration in logback.

[tscroggins]

Hi @priyanka2887,

At which layer? TLS? HTTP? Splunk?

TLS compression is largely deprecated, vulnerable to well-known attacks, and not (as far as I know) available in core JDK implementations of TLS 1.2+.

HttpEventCollectorLogbackAppender's underlying HTTP implementation, OkHttp, should compress any payload over 1024 bytes by default. See https://github.com/square/okhttp/blob/master/okhttp/src/main/kotlin/okhttp3/OkHttpClient.kt.

HttpEventCollectorLogbackAppender doesn't expose a method or property to modify the threshold. See https://github.com/splunk/splunk-library-javalogging/blob/main/src/main/java/com/splunk/logging/Http... and https://github.com/splunk/splunk-library-javalogging/blob/main/src/main/java/com/splunk/logging/Http....

If you want to add support for modifying the compression threshold, see the Contributing section at https://github.com/splunk/splunk-library-javalogging/blob/main/README.md. 

Raw data is always compressed in Splunk, although the algorithm is configurable. See the journalCompression setting in https://docs.splunk.com/Documentation/Splunk/latest/Admin/Indexesconf.