Getting Data In

Props.conf not picking up linemerge

markhvesta
Path Finder

Lines in my sourcetype are not being picked up correctly at all.  Each event is being split into dozens of lines.  Also, when I go into the Settings in the UI for sourcetypes, I see all of the configs matching what I have set except for SHOULD_LINEMERGE = true.  This comes up as false.  I try resetting it in the UI and it still comes up as false even though that should not be set anywhere.  Btool shows it should be set to true, but it still comes up as false.

Btool shows these settings

[kube:container:applicationservice-app]
BREAK_ONLY_BEFORE_DATE = true
LINE_BREAKER = (\d{2}\:\d{2}\:\d{2}\.\d{3})(?:\s\[Thread)
MAX_TIMESTAMP_LOOKAHEAD = 128
SHOULD_LINEMERGE = true
TIME_FORMAT = %H:%M:%S.%Q

 

Labels (1)
0 Karma
1 Solution

mattymo
Splunk Employee
Splunk Employee

Hey @markhvesta 

This is because Splunk Connect for Kubernetes sends data using the HTTP Event Collector ("HEC") Event  endpoint and events that come through "HEC" event endpoint do not hit the line merge processor.

For more info, the events are sent like this to automate extraction of key Kubernetes metadata for you. 

To deal with multiline events, the line merge must be done ahead of time in the logging collector config:

https://github.com/splunk/splunk-connect-for-kubernetes/blob/09cb0462a624d348aa6bc94c0996599907de88f...

Like this example that I applied to the Connect for Kubernetes logging pod:

 

  logs:
    sck:
      from:
        pod: sck-splunk-kubernetes-
        container: splunk-fluentd-k8s-
      multiline:
        firstline: /^\d{4}-\d{2}-\d{2}\s\d{2}\:\d{2}\:\d{2}\s\+\d{4}\s\[\w+\]\:/
        separator: "\n"

 

 This is done with the fluentd "concat" filter that we ship in Splunk Connect for Kubernetes. 

Be sure to use rubular.com to test your regex as Fluentd uses ruby regex. 

Similar option is available in our OpenTelemetry collector, which you may also want to get familiar with in the future as it is a more performant option for k8s log collection if you need high velocity logging as your clusters get bigger and bigger. 

https://github.com/signalfx/splunk-otel-collector-chart/blob/26e37677a947d7081d686d2b7533057196bba07...

 

- MattyMo

View solution in original post

mattymo
Splunk Employee
Splunk Employee

Hey @markhvesta 

This is because Splunk Connect for Kubernetes sends data using the HTTP Event Collector ("HEC") Event  endpoint and events that come through "HEC" event endpoint do not hit the line merge processor.

For more info, the events are sent like this to automate extraction of key Kubernetes metadata for you. 

To deal with multiline events, the line merge must be done ahead of time in the logging collector config:

https://github.com/splunk/splunk-connect-for-kubernetes/blob/09cb0462a624d348aa6bc94c0996599907de88f...

Like this example that I applied to the Connect for Kubernetes logging pod:

 

  logs:
    sck:
      from:
        pod: sck-splunk-kubernetes-
        container: splunk-fluentd-k8s-
      multiline:
        firstline: /^\d{4}-\d{2}-\d{2}\s\d{2}\:\d{2}\:\d{2}\s\+\d{4}\s\[\w+\]\:/
        separator: "\n"

 

 This is done with the fluentd "concat" filter that we ship in Splunk Connect for Kubernetes. 

Be sure to use rubular.com to test your regex as Fluentd uses ruby regex. 

Similar option is available in our OpenTelemetry collector, which you may also want to get familiar with in the future as it is a more performant option for k8s log collection if you need high velocity logging as your clusters get bigger and bigger. 

https://github.com/signalfx/splunk-otel-collector-chart/blob/26e37677a947d7081d686d2b7533057196bba07...

 

- MattyMo

markhvesta
Path Finder

23:08:13.182 [Thread-440] INFO io.vesta.vnext.logging.azure.blob.AzureLogFile - [INFORMATIONAL][d811ffd5-72eb-4444-af55-6f5d002c95d0] {
"date" : "2021-12-14T23:08:13.182419",
"correlationId" : "d811ffd5-72eb-4444-af55-6f5d002c95d0",
"logLevel" : "INFORMATIONAL",
"category" : "HttpResponse",
"requestPath" : "/api/domaindata/find/PaymentProcessors/search",
"azureUserId" : "e8668b6a-57f7-474d-8e67-3df5bae5c55c",
"customerId" : 87,
"message" : "[{\"PaymentProcessorId

0 Karma

isoutamo
SplunkTrust
SplunkTrust
Is this a full event? It seems that there is something missing on message part?

markhvesta
Path Finder

It isn't the full event; some of these events can be fairly verbose. but in this example each line or every other line would be its own event.

0 Karma

isoutamo
SplunkTrust
SplunkTrust
Can you add the last “line” of this event, so we could see how it ends?

isoutamo
SplunkTrust
SplunkTrust
Can you sent some sample events?
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...