topic How do I filter out DEBUG entries from a linux / Unix logfile with the heavy forwarder? in Splunk Search

How do I filter out DEBUG entries from a linux / Unix logfile with the heavy forwarder?

gozulin — Thu, 30 Jan 2014 22:26:39 GMT

We're having some licensing violations when we need to turn on DEBUG on some of our services and we'd like to just have a regex nullqueue any debug entries before forwarding them to the indexers.

Re: How do I filter out DEBUG entries from a linux / Unix logfile with the heavy forwarder?

David — Fri, 31 Jan 2014 00:13:26 GMT

You should be able to follow the guidance of this answers post but replace the regex with DEBUG. You could make the regex more specific by providing a few example logs (e.g., LogLevel DEBUG if that's what your logs look like).

Re: How do I filter out DEBUG entries from a linux / Unix logfile with the heavy forwarder?

gozulin — Mon, 28 Sep 2020 15:46:53 GMT

Using the previous answer, here is what worked to filter out DEBUG messages:

in props.conf:

TRANSFORMS-null= setnull
[mysourcetype]
NO_BINARY_CHECK = 1
pulldown_type = 1

In transforms.conf:

[setnull]
REGEX = [DEBUG]
DEST_KEY = queue
FORMAT = nullQueue

Re: How do I filter out DEBUG entries from a linux / Unix logfile with the heavy forwarder?

David — Fri, 31 Jan 2014 05:53:22 GMT

Have you verified that this is not matching more than you intended? In regex terms, that should match anything with a capital D, E, B, U, or G.

Re: How do I filter out DEBUG entries from a linux / Unix logfile with the heavy forwarder?

gozulin — Fri, 31 Jan 2014 17:42:40 GMT

Huh, you're right of course. It's weird because the content of the file actually has backslashes in it. Not sure why they didn't show up!

[setnull]
REGEX = \[DEBUG\]
DEST_KEY = queue
FORMAT = nullQueue

Re: How do I filter out DEBUG entries from a linux / Unix logfile with the heavy forwarder?

mukherjee_mk — Tue, 29 Sep 2020 08:10:05 GMT

Thanks for your help folks. I notice that we have to keep these rows in the right order though. The name of the sourcetype should be at the beginning of the segment.

in props.conf (notice that the sourcetype is the first line of the segment):
[mysourcetype]
TRANSFORMS-null= setnull
NO_BINARY_CHECK = 1
pulldown_type = 1

In transforms.conf:
[setnull]
REGEX = DEBUG
DEST_KEY = queue
FORMAT = nullQueue