topic Re: Props.conf not picking up linemerge in Getting Data In

Props.conf not picking up linemerge

markhvesta — Fri, 17 Dec 2021 17:53:48 GMT

Lines in my sourcetype are not being picked up correctly at all. Each event is being split into dozens of lines. Also, when I go into the Settings in the UI for sourcetypes, I see all of the configs matching what I have set except for SHOULD_LINEMERGE = true. This comes up as false. I try resetting it in the UI and it still comes up as false even though that should not be set anywhere. Btool shows it should be set to true, but it still comes up as false.

Btool shows these settings

[kube:container:applicationservice-app]
BREAK_ONLY_BEFORE_DATE = true
LINE_BREAKER = (\d{2}\:\d{2}\:\d{2}\.\d{3})(?:\s\[Thread)
MAX_TIMESTAMP_LOOKAHEAD = 128
SHOULD_LINEMERGE = true
TIME_FORMAT = %H:%M:%S.%Q

Re: Props.conf not picking up linemerge

isoutamo — Fri, 17 Dec 2021 18:17:49 GMT

Can you sent some sample events?

Re: Props.conf not picking up linemerge

markhvesta — Fri, 17 Dec 2021 18:53:30 GMT

23:08:13.182 [Thread-440] INFO io.vesta.vnext.logging.azure.blob.AzureLogFile - [INFORMATIONAL][d811ffd5-72eb-4444-af55-6f5d002c95d0] {
"date" : "2021-12-14T23:08:13.182419",
"correlationId" : "d811ffd5-72eb-4444-af55-6f5d002c95d0",
"logLevel" : "INFORMATIONAL",
"category" : "HttpResponse",
"requestPath" : "/api/domaindata/find/PaymentProcessors/search",
"azureUserId" : "e8668b6a-57f7-474d-8e67-3df5bae5c55c",
"customerId" : 87,
"message" : "[{\"PaymentProcessorId

Re: Props.conf not picking up linemerge

isoutamo — Fri, 17 Dec 2021 21:09:30 GMT

Is this a full event? It seems that there is something missing on message part?

Re: Props.conf not picking up linemerge

markhvesta — Fri, 17 Dec 2021 21:14:14 GMT

It isn't the full event; some of these events can be fairly verbose. but in this example each line or every other line would be its own event.

Re: Props.conf not picking up linemerge

isoutamo — Sat, 18 Dec 2021 11:36:08 GMT

Can you add the last “line” of this event, so we could see how it ends?

Re: Props.conf not picking up linemerge

mattymo — Sat, 18 Dec 2021 14:05:17 GMT

Hey @markhvesta

This is because Splunk Connect for Kubernetes sends data using the HTTP Event Collector ("HEC") Event endpoint and events that come through "HEC" event endpoint do not hit the line merge processor.

For more info, the events are sent like this to automate extraction of key Kubernetes metadata for you.

To deal with multiline events, the line merge must be done ahead of time in the logging collector config:

https://github.com/splunk/splunk-connect-for-kubernetes/blob/09cb0462a624d348aa6bc94c0996599907de88f7/helm-chart/splunk-connect-for-kubernetes/values.yaml#L217-L285

Like this example that I applied to the Connect for Kubernetes logging pod:

logs: sck: from: pod: sck-splunk-kubernetes- container: splunk-fluentd-k8s- multiline: firstline: /^\d{4}-\d{2}-\d{2}\s\d{2}\:\d{2}\:\d{2}\s\+\d{4}\s\[\w+\]\:/ separator: "\n"

This is done with the fluentd "concat" filter that we ship in Splunk Connect for Kubernetes.

Be sure to use rubular.com to test your regex as Fluentd uses ruby regex.

Similar option is available in our OpenTelemetry collector, which you may also want to get familiar with in the future as it is a more performant option for k8s log collection if you need high velocity logging as your clusters get bigger and bigger.

https://github.com/signalfx/splunk-otel-collector-chart/blob/26e37677a947d7081d686d2b7533057196bba070/helm-charts/splunk-otel-collector/values.yaml#L369-L399