Splunk Search

How do I filter out DEBUG entries from a linux / Unix logfile with the heavy forwarder?

Communicator

We're having some licensing violations when we need to turn on DEBUG on some of our services and we'd like to just have a regex nullqueue any debug entries before forwarding them to the indexers.

0 Karma
1 Solution

Communicator

Using the previous answer, here is what worked to filter out DEBUG messages:

in props.conf:

TRANSFORMS-null= setnull
[mysourcetype]
NO_BINARY_CHECK = 1
pulldown_type = 1

In transforms.conf:

[setnull]
REGEX = [DEBUG]
DEST_KEY = queue
FORMAT = nullQueue

View solution in original post

0 Karma

Explorer

Thanks for your help folks. I notice that we have to keep these rows in the right order though. The name of the sourcetype should be at the beginning of the segment.

in props.conf (notice that the sourcetype is the first line of the segment):
[mysourcetype]
TRANSFORMS-null= setnull
NO_BINARY_CHECK = 1
pulldown_type = 1

In transforms.conf:
[setnull]
REGEX = DEBUG
DEST_KEY = queue
FORMAT = nullQueue

0 Karma

Communicator

Using the previous answer, here is what worked to filter out DEBUG messages:

in props.conf:

TRANSFORMS-null= setnull
[mysourcetype]
NO_BINARY_CHECK = 1
pulldown_type = 1

In transforms.conf:

[setnull]
REGEX = [DEBUG]
DEST_KEY = queue
FORMAT = nullQueue

View solution in original post

0 Karma

Communicator

Huh, you're right of course. It's weird because the content of the file actually has backslashes in it. Not sure why they didn't show up!

[setnull]
REGEX = \[DEBUG\]
DEST_KEY = queue
FORMAT = nullQueue

0 Karma

Splunk Employee
Splunk Employee

Have you verified that this is not matching more than you intended? In regex terms, that should match anything with a capital D, E, B, U, or G.

0 Karma

Splunk Employee
Splunk Employee

You should be able to follow the guidance of this answers post but replace the regex with DEBUG. You could make the regex more specific by providing a few example logs (e.g., LogLevel DEBUG if that's what your logs look like).