Getting Data In

Props.conf not picking up linemerge

markhvesta
Path Finder

Lines in my sourcetype are not being picked up correctly at all.  Each event is being split into dozens of lines.  Also, when I go into the Settings in the UI for sourcetypes, I see all of the configs matching what I have set except for SHOULD_LINEMERGE = true.  This comes up as false.  I try resetting it in the UI and it still comes up as false even though that should not be set anywhere.  Btool shows it should be set to true, but it still comes up as false.

Btool shows these settings

[kube:container:applicationservice-app]
BREAK_ONLY_BEFORE_DATE = true
LINE_BREAKER = (\d{2}\:\d{2}\:\d{2}\.\d{3})(?:\s\[Thread)
MAX_TIMESTAMP_LOOKAHEAD = 128
SHOULD_LINEMERGE = true
TIME_FORMAT = %H:%M:%S.%Q

 

Labels (1)
0 Karma
1 Solution

mattymo
Splunk Employee
Splunk Employee

Hey @markhvesta 

This is because Splunk Connect for Kubernetes sends data using the HTTP Event Collector ("HEC") Event  endpoint and events that come through "HEC" event endpoint do not hit the line merge processor.

For more info, the events are sent like this to automate extraction of key Kubernetes metadata for you. 

To deal with multiline events, the line merge must be done ahead of time in the logging collector config:

https://github.com/splunk/splunk-connect-for-kubernetes/blob/09cb0462a624d348aa6bc94c0996599907de88f...

Like this example that I applied to the Connect for Kubernetes logging pod:

 

  logs:
    sck:
      from:
        pod: sck-splunk-kubernetes-
        container: splunk-fluentd-k8s-
      multiline:
        firstline: /^\d{4}-\d{2}-\d{2}\s\d{2}\:\d{2}\:\d{2}\s\+\d{4}\s\[\w+\]\:/
        separator: "\n"

 

 This is done with the fluentd "concat" filter that we ship in Splunk Connect for Kubernetes. 

Be sure to use rubular.com to test your regex as Fluentd uses ruby regex. 

Similar option is available in our OpenTelemetry collector, which you may also want to get familiar with in the future as it is a more performant option for k8s log collection if you need high velocity logging as your clusters get bigger and bigger. 

https://github.com/signalfx/splunk-otel-collector-chart/blob/26e37677a947d7081d686d2b7533057196bba07...

 

- MattyMo

View solution in original post

mattymo
Splunk Employee
Splunk Employee

Hey @markhvesta 

This is because Splunk Connect for Kubernetes sends data using the HTTP Event Collector ("HEC") Event  endpoint and events that come through "HEC" event endpoint do not hit the line merge processor.

For more info, the events are sent like this to automate extraction of key Kubernetes metadata for you. 

To deal with multiline events, the line merge must be done ahead of time in the logging collector config:

https://github.com/splunk/splunk-connect-for-kubernetes/blob/09cb0462a624d348aa6bc94c0996599907de88f...

Like this example that I applied to the Connect for Kubernetes logging pod:

 

  logs:
    sck:
      from:
        pod: sck-splunk-kubernetes-
        container: splunk-fluentd-k8s-
      multiline:
        firstline: /^\d{4}-\d{2}-\d{2}\s\d{2}\:\d{2}\:\d{2}\s\+\d{4}\s\[\w+\]\:/
        separator: "\n"

 

 This is done with the fluentd "concat" filter that we ship in Splunk Connect for Kubernetes. 

Be sure to use rubular.com to test your regex as Fluentd uses ruby regex. 

Similar option is available in our OpenTelemetry collector, which you may also want to get familiar with in the future as it is a more performant option for k8s log collection if you need high velocity logging as your clusters get bigger and bigger. 

https://github.com/signalfx/splunk-otel-collector-chart/blob/26e37677a947d7081d686d2b7533057196bba07...

 

- MattyMo

markhvesta
Path Finder

23:08:13.182 [Thread-440] INFO io.vesta.vnext.logging.azure.blob.AzureLogFile - [INFORMATIONAL][d811ffd5-72eb-4444-af55-6f5d002c95d0] {
"date" : "2021-12-14T23:08:13.182419",
"correlationId" : "d811ffd5-72eb-4444-af55-6f5d002c95d0",
"logLevel" : "INFORMATIONAL",
"category" : "HttpResponse",
"requestPath" : "/api/domaindata/find/PaymentProcessors/search",
"azureUserId" : "e8668b6a-57f7-474d-8e67-3df5bae5c55c",
"customerId" : 87,
"message" : "[{\"PaymentProcessorId

0 Karma

isoutamo
SplunkTrust
SplunkTrust
Is this a full event? It seems that there is something missing on message part?

markhvesta
Path Finder

It isn't the full event; some of these events can be fairly verbose. but in this example each line or every other line would be its own event.

0 Karma

isoutamo
SplunkTrust
SplunkTrust
Can you add the last “line” of this event, so we could see how it ends?

isoutamo
SplunkTrust
SplunkTrust
Can you sent some sample events?
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...