topic SHOULD_LINEMERGE from HEC props.conf in Splunk Enterprise

SHOULD_LINEMERGE from HEC props.conf

robertosegantin — Wed, 28 Oct 2020 17:01:49 GMT

On Kubernetes environment there is installed Fluentd Splunk plugin which sends to Heavy Forwarder, via HEC, the standard output application logs.

The standard output application logs are not structured and I'm not able to apply line merge to them.

My input.conf is:

[http://k8s_hec]
disabled = 0
index = em_events
source = em_metrics
token = aaaaaaaa-bbbb-cccc-dddd-fffffffffff

Fluentd defined many sourcetypes, and all custom applications sourcetypes end with "app"; for example:

kube:container:goofy-app
kube:container:donald-duck-app

So I defined these two configurations in props.conf inside my HF, but I'm not able to merge events:

[kube:container:*-app]
SHOULD_LINEMERGE=true
LINE_BREAKER=([\r\n]+)
NO_BINARY_CHECK=true
CHARSET=UTF-8
MAX_TIMESTAMP_LOOKAHEAD=30
disabled=false
TIME_FORMAT=%Y-%m-%d %H:%M:%S.%3N
TIME_PREFIX=^
MAX_EVENTS=1024

[source::k8s_hec]
SHOULD_LINEMERGE=true
LINE_BREAKER=([\r\n]+)
NO_BINARY_CHECK=true
CHARSET=UTF-8
MAX_TIMESTAMP_LOOKAHEAD=30
disabled=false
TIME_FORMAT=%Y-%m-%d %H:%M:%S.%3N
TIME_PREFIX=^
MAX_EVENTS=1024

Someone can help me?

Re: SHOULD_LINEMERGE from HEC props.conf

richgalloway — Wed, 28 Oct 2020 19:34:03 GMT

Have you looked at the indexed data to see if it's arriving in the expected format?

Re: SHOULD_LINEMERGE from HEC props.conf

robertosegantin — Thu, 29 Oct 2020 08:59:05 GMT

To be more clear I update the events indexed:

I need to merge all these events received via HEC.

If I read these events in a classical way, via log reading, with a more simple configuration, Splunk is able to merge them:

I hope that is not a limit about HEC method.

Thanks

Re: SHOULD_LINEMERGE from HEC props.conf

maraman_splunk — Thu, 29 Oct 2020 10:17:03 GMT

humm, your props doesnt really tell how to merge.

One possible solution would be to add date form at the end of your LINE_BREAKER

Re: SHOULD_LINEMERGE from HEC props.conf

mattymo — Sun, 28 Feb 2021 14:27:49 GMT

you need to use the concat filter to line merge these events BEFORE HEC. Please see multiline filter rules in Splunk Connect for Kubernetes

https://github.com/splunk/splunk-connect-for-kubernetes#processing-multi-line-logs

Concat filter plugin is used and make sure the HEC payload makes it to Splunk already line-merged.

There are gnarly props and transforms hack to do this work but it is better at the collector or in Data Stream Processor or other stream proc products out there. This is because the container runtimes themselves don't even support multiline logging at this point