How to handle multiline events from k8s via opentelemetry / HWF

[rk99]

Hi - we have been sending data from our K8s cluster to splunk hwf which then forwards to the indexer.  It works great except for multiline events - particularly stack traces.

I have setup the sourcetype and validated it line breaks correctly with the data inputs file input gui.  I've got the sourcetype defined in a props on the hwf and can choose it from the drop down in the gui and it does the line breaks.  I also have a transform that adds a new field so I know its hitting the props.

However i'm still only getting one line per event.  I'm wondering if there's something that needs to be set in opentelemetry to configure the line breaking before it goes to the hwf.

if it helps this is the sourcetype

[kube:container:test-app]
LINE_BREAKER = ([\r\n]+)\w{3} \d{4}
NO_BINARY_CHECK = true
category = Custom
pulldown_type = true
BREAK_ONLY_BEFORE_DATE =
DATETIME_CONFIG =
MAX_TIMESTAMP_LOOKAHEAD = 25
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%d %H:%M:%S
TRUNCATE = 0

Thank you for any help

[livehybrid · Accepted solution]

Hi @rk99 

Are you using the Splunk OTel collector for Kubernetes (SOCK)? If so the following should help https://community.splunk.com/t5/Community-Blog/Combine-Multiline-Logs-into-a-Single-Event-with-SOCK-... as it walks through how to achieve multiline events into Splunk.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[rk99]

thanks @livehybrid  - the documentation was just what we needed, we've got the multilines working perfectly now.

for anyone else this is what we added (the multilineConfigs) for log4j that started lines with YYYY-MM-DD

spec:
  values:
    logCollection:
      containers:
        multilineConfigs:
          - namespaceName:
              value: <NS>
            podName:
              value: <podName>-*
              useRegexp: true
            firstEntryRegex: ^\d{4}\-\d{2}\-\d{2}\s

[rk99]

Yes, thats what we're using - thank you, i'll check this today