Props.conf not picking up linemerge

[markhvesta]

Lines in my sourcetype are not being picked up correctly at all.  Each event is being split into dozens of lines.  Also, when I go into the Settings in the UI for sourcetypes, I see all of the configs matching what I have set except for SHOULD_LINEMERGE = true.  This comes up as false.  I try resetting it in the UI and it still comes up as false even though that should not be set anywhere.  Btool shows it should be set to true, but it still comes up as false.

Btool shows these settings

[kube:container:applicationservice-app]
BREAK_ONLY_BEFORE_DATE = true
LINE_BREAKER = (\d{2}\:\d{2}\:\d{2}\.\d{3})(?:\s\[Thread)
MAX_TIMESTAMP_LOOKAHEAD = 128
SHOULD_LINEMERGE = true
TIME_FORMAT = %H:%M:%S.%Q

[mattymo · Accepted solution]

Hey @markhvesta 

This is because Splunk Connect for Kubernetes sends data using the HTTP Event Collector ("HEC") Event  endpoint and events that come through "HEC" event endpoint do not hit the line merge processor.

For more info, the events are sent like this to automate extraction of key Kubernetes metadata for you. 

To deal with multiline events, the line merge must be done ahead of time in the logging collector config:

https://github.com/splunk/splunk-connect-for-kubernetes/blob/09cb0462a624d348aa6bc94c0996599907de88f...

Like this example that I applied to the Connect for Kubernetes logging pod:

  logs:
    sck:
      from:
        pod: sck-splunk-kubernetes-
        container: splunk-fluentd-k8s-
      multiline:
        firstline: /^\d{4}-\d{2}-\d{2}\s\d{2}\:\d{2}\:\d{2}\s\+\d{4}\s\[\w+\]\:/
        separator: "\n"

 This is done with the fluentd "concat" filter that we ship in Splunk Connect for Kubernetes. 

Be sure to use rubular.com to test your regex as Fluentd uses ruby regex. 

Similar option is available in our OpenTelemetry collector, which you may also want to get familiar with in the future as it is a more performant option for k8s log collection if you need high velocity logging as your clusters get bigger and bigger. 

https://github.com/signalfx/splunk-otel-collector-chart/blob/26e37677a947d7081d686d2b7533057196bba07...

[markhvesta]

23:08:13.182 [Thread-440] INFO io.vesta.vnext.logging.azure.blob.AzureLogFile - [INFORMATIONAL][d811ffd5-72eb-4444-af55-6f5d002c95d0] {
"date" : "2021-12-14T23:08:13.182419",
"correlationId" : "d811ffd5-72eb-4444-af55-6f5d002c95d0",
"logLevel" : "INFORMATIONAL",
"category" : "HttpResponse",
"requestPath" : "/api/domaindata/find/PaymentProcessors/search",
"azureUserId" : "e8668b6a-57f7-474d-8e67-3df5bae5c55c",
"customerId" : 87,
"message" : "[{\"PaymentProcessorId

[isoutamo]

Is this a full event? It seems that there is something missing on message part?

[markhvesta]

It isn't the full event; some of these events can be fairly verbose. but in this example each line or every other line would be its own event.

[isoutamo]

Can you add the last “line” of this event, so we could see how it ends?

[isoutamo]

Can you sent some sample events?