Getting Data In

Props.conf not picking up linemerge

markhvesta
Path Finder

Lines in my sourcetype are not being picked up correctly at all.  Each event is being split into dozens of lines.  Also, when I go into the Settings in the UI for sourcetypes, I see all of the configs matching what I have set except for SHOULD_LINEMERGE = true.  This comes up as false.  I try resetting it in the UI and it still comes up as false even though that should not be set anywhere.  Btool shows it should be set to true, but it still comes up as false.

Btool shows these settings

[kube:container:applicationservice-app]
BREAK_ONLY_BEFORE_DATE = true
LINE_BREAKER = (\d{2}\:\d{2}\:\d{2}\.\d{3})(?:\s\[Thread)
MAX_TIMESTAMP_LOOKAHEAD = 128
SHOULD_LINEMERGE = true
TIME_FORMAT = %H:%M:%S.%Q

 

Labels (1)
0 Karma
1 Solution

mattymo
Splunk Employee
Splunk Employee

Hey @markhvesta 

This is because Splunk Connect for Kubernetes sends data using the HTTP Event Collector ("HEC") Event  endpoint and events that come through "HEC" event endpoint do not hit the line merge processor.

For more info, the events are sent like this to automate extraction of key Kubernetes metadata for you. 

To deal with multiline events, the line merge must be done ahead of time in the logging collector config:

https://github.com/splunk/splunk-connect-for-kubernetes/blob/09cb0462a624d348aa6bc94c0996599907de88f...

Like this example that I applied to the Connect for Kubernetes logging pod:

 

  logs:
    sck:
      from:
        pod: sck-splunk-kubernetes-
        container: splunk-fluentd-k8s-
      multiline:
        firstline: /^\d{4}-\d{2}-\d{2}\s\d{2}\:\d{2}\:\d{2}\s\+\d{4}\s\[\w+\]\:/
        separator: "\n"

 

 This is done with the fluentd "concat" filter that we ship in Splunk Connect for Kubernetes. 

Be sure to use rubular.com to test your regex as Fluentd uses ruby regex. 

Similar option is available in our OpenTelemetry collector, which you may also want to get familiar with in the future as it is a more performant option for k8s log collection if you need high velocity logging as your clusters get bigger and bigger. 

https://github.com/signalfx/splunk-otel-collector-chart/blob/26e37677a947d7081d686d2b7533057196bba07...

 

- MattyMo

View solution in original post

mattymo
Splunk Employee
Splunk Employee

Hey @markhvesta 

This is because Splunk Connect for Kubernetes sends data using the HTTP Event Collector ("HEC") Event  endpoint and events that come through "HEC" event endpoint do not hit the line merge processor.

For more info, the events are sent like this to automate extraction of key Kubernetes metadata for you. 

To deal with multiline events, the line merge must be done ahead of time in the logging collector config:

https://github.com/splunk/splunk-connect-for-kubernetes/blob/09cb0462a624d348aa6bc94c0996599907de88f...

Like this example that I applied to the Connect for Kubernetes logging pod:

 

  logs:
    sck:
      from:
        pod: sck-splunk-kubernetes-
        container: splunk-fluentd-k8s-
      multiline:
        firstline: /^\d{4}-\d{2}-\d{2}\s\d{2}\:\d{2}\:\d{2}\s\+\d{4}\s\[\w+\]\:/
        separator: "\n"

 

 This is done with the fluentd "concat" filter that we ship in Splunk Connect for Kubernetes. 

Be sure to use rubular.com to test your regex as Fluentd uses ruby regex. 

Similar option is available in our OpenTelemetry collector, which you may also want to get familiar with in the future as it is a more performant option for k8s log collection if you need high velocity logging as your clusters get bigger and bigger. 

https://github.com/signalfx/splunk-otel-collector-chart/blob/26e37677a947d7081d686d2b7533057196bba07...

 

- MattyMo

markhvesta
Path Finder

23:08:13.182 [Thread-440] INFO io.vesta.vnext.logging.azure.blob.AzureLogFile - [INFORMATIONAL][d811ffd5-72eb-4444-af55-6f5d002c95d0] {
"date" : "2021-12-14T23:08:13.182419",
"correlationId" : "d811ffd5-72eb-4444-af55-6f5d002c95d0",
"logLevel" : "INFORMATIONAL",
"category" : "HttpResponse",
"requestPath" : "/api/domaindata/find/PaymentProcessors/search",
"azureUserId" : "e8668b6a-57f7-474d-8e67-3df5bae5c55c",
"customerId" : 87,
"message" : "[{\"PaymentProcessorId

0 Karma

isoutamo
SplunkTrust
SplunkTrust
Is this a full event? It seems that there is something missing on message part?

markhvesta
Path Finder

It isn't the full event; some of these events can be fairly verbose. but in this example each line or every other line would be its own event.

0 Karma

isoutamo
SplunkTrust
SplunkTrust
Can you add the last “line” of this event, so we could see how it ends?

isoutamo
SplunkTrust
SplunkTrust
Can you sent some sample events?
Get Updates on the Splunk Community!

What’s new on Splunk Lantern in August

This month’s Splunk Lantern update gives you the low-down on all of the articles we’ve published over the past ...

Welcome to the Future of Data Search & Exploration

You have more data coming at you than ever before. Over the next five years, the total amount of digital data ...

This Week's Community Digest - Splunk Community Happenings [8.3.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...