Splunk Enterprise

SHOULD_LINEMERGE from HEC props.conf

robertosegantin
Path Finder

On Kubernetes environment there is installed Fluentd Splunk plugin which sends to Heavy Forwarder, via HEC, the standard output application logs.

The standard output application logs are not structured and I'm not able to  apply line merge to them.

My input.conf is:

[http://k8s_hec]
disabled = 0
index = em_events
source = em_metrics
token = aaaaaaaa-bbbb-cccc-dddd-fffffffffff

 

 

 

Fluentd defined many sourcetypes, and all custom applications sourcetypes end with "app";  for example: 

  • kube:container:goofy-app
  • kube:container:donald-duck-app

 

So I defined these two configurations in props.conf inside my HF, but I'm not able to merge events:

[kube:container:*-app]
SHOULD_LINEMERGE=true
LINE_BREAKER=([\r\n]+)
NO_BINARY_CHECK=true
CHARSET=UTF-8
MAX_TIMESTAMP_LOOKAHEAD=30
disabled=false
TIME_FORMAT=%Y-%m-%d %H:%M:%S.%3N
TIME_PREFIX=^
MAX_EVENTS=1024

[source::k8s_hec]
SHOULD_LINEMERGE=true
LINE_BREAKER=([\r\n]+)
NO_BINARY_CHECK=true
CHARSET=UTF-8
MAX_TIMESTAMP_LOOKAHEAD=30
disabled=false
TIME_FORMAT=%Y-%m-%d %H:%M:%S.%3N
TIME_PREFIX=^
MAX_EVENTS=1024

 

 

 

Someone can help me?

Labels (1)

mattymo
Splunk Employee
Splunk Employee

you need to use the concat filter to line merge these events BEFORE HEC. Please see multiline filter rules in Splunk Connect for Kubernetes

https://github.com/splunk/splunk-connect-for-kubernetes#processing-multi-line-logs

Concat filter plugin is used and make sure the HEC payload makes it to Splunk already line-merged. 

There are gnarly props and transforms hack to do this work but it is better at the collector or in Data Stream Processor or other stream proc products out there. This is because the container runtimes themselves don't even support multiline logging at this point

0 Karma

maraman_splunk
Splunk Employee
Splunk Employee

humm, your props doesnt really tell how to merge.

One possible solution would be to add date form at the end of your LINE_BREAKER

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Have you looked at the indexed data to see if it's arriving in the expected format?

 

---
If this reply helps you, an upvote would be appreciated.
0 Karma

robertosegantin
Path Finder

To be more clear I update the events indexed:

indexed_data.png

I need to merge all these events received via HEC.

If I read these events in a classical way, via log reading, with a more simple configuration, Splunk is able to merge them:

adddata.png

 

I hope that is not a limit about HEC method.

Thanks