SHOULD_LINEMERGE from HEC props.conf

[robertosegantin]

On Kubernetes environment there is installed Fluentd Splunk plugin which sends to Heavy Forwarder, via HEC, the standard output application logs.

The standard output application logs are not structured and I'm not able to  apply line merge to them.

My input.conf is:

[http://k8s_hec]
disabled = 0
index = em_events
source = em_metrics
token = aaaaaaaa-bbbb-cccc-dddd-fffffffffff

Fluentd defined many sourcetypes, and all custom applications sourcetypes end with "app";  for example: 

• kube:container:goofy-app
• kube:container:donald-duck-app

So I defined these two configurations in props.conf inside my HF, but I'm not able to merge events:

[kube:container:*-app]
SHOULD_LINEMERGE=true
LINE_BREAKER=([\r\n]+)
NO_BINARY_CHECK=true
CHARSET=UTF-8
MAX_TIMESTAMP_LOOKAHEAD=30
disabled=false
TIME_FORMAT=%Y-%m-%d %H:%M:%S.%3N
TIME_PREFIX=^
MAX_EVENTS=1024

[source::k8s_hec]
SHOULD_LINEMERGE=true
LINE_BREAKER=([\r\n]+)
NO_BINARY_CHECK=true
CHARSET=UTF-8
MAX_TIMESTAMP_LOOKAHEAD=30
disabled=false
TIME_FORMAT=%Y-%m-%d %H:%M:%S.%3N
TIME_PREFIX=^
MAX_EVENTS=1024

Someone can help me?

[mattymo]

you need to use the concat filter to line merge these events BEFORE HEC. Please see multiline filter rules in Splunk Connect for Kubernetes

https://github.com/splunk/splunk-connect-for-kubernetes#processing-multi-line-logs

Concat filter plugin is used and make sure the HEC payload makes it to Splunk already line-merged. 

There are gnarly props and transforms hack to do this work but it is better at the collector or in Data Stream Processor or other stream proc products out there. This is because the container runtimes themselves don't even support multiline logging at this point

[maraman_splunk]

humm, your props doesnt really tell how to merge.

One possible solution would be to add date form at the end of your LINE_BREAKER

[richgalloway]

Have you looked at the indexed data to see if it's arriving in the expected format?

[robertosegantin]

To be more clear I update the events indexed:

I need to merge all these events received via HEC.

If I read these events in a classical way, via log reading, with a more simple configuration, Splunk is able to merge them:

I hope that is not a limit about HEC method.

Thanks