Splunk Search

How do I filter out DEBUG entries from a linux / Unix logfile with the heavy forwarder?

gozulin
Communicator

We're having some licensing violations when we need to turn on DEBUG on some of our services and we'd like to just have a regex nullqueue any debug entries before forwarding them to the indexers.

0 Karma
1 Solution

gozulin
Communicator

Using the previous answer, here is what worked to filter out DEBUG messages:

in props.conf:

TRANSFORMS-null= setnull
[mysourcetype]
NO_BINARY_CHECK = 1
pulldown_type = 1

In transforms.conf:

[setnull]
REGEX = [DEBUG]
DEST_KEY = queue
FORMAT = nullQueue

View solution in original post

0 Karma

mukherjee_mk
Explorer

Thanks for your help folks. I notice that we have to keep these rows in the right order though. The name of the sourcetype should be at the beginning of the segment.

in props.conf (notice that the sourcetype is the first line of the segment):
[mysourcetype]
TRANSFORMS-null= setnull
NO_BINARY_CHECK = 1
pulldown_type = 1

In transforms.conf:
[setnull]
REGEX = DEBUG
DEST_KEY = queue
FORMAT = nullQueue

0 Karma

gozulin
Communicator

Using the previous answer, here is what worked to filter out DEBUG messages:

in props.conf:

TRANSFORMS-null= setnull
[mysourcetype]
NO_BINARY_CHECK = 1
pulldown_type = 1

In transforms.conf:

[setnull]
REGEX = [DEBUG]
DEST_KEY = queue
FORMAT = nullQueue

0 Karma

gozulin
Communicator

Huh, you're right of course. It's weird because the content of the file actually has backslashes in it. Not sure why they didn't show up!

[setnull]
REGEX = \[DEBUG\]
DEST_KEY = queue
FORMAT = nullQueue

0 Karma

David
Splunk Employee
Splunk Employee

Have you verified that this is not matching more than you intended? In regex terms, that should match anything with a capital D, E, B, U, or G.

0 Karma

David
Splunk Employee
Splunk Employee

You should be able to follow the guidance of this answers post but replace the regex with DEBUG. You could make the regex more specific by providing a few example logs (e.g., LogLevel DEBUG if that's what your logs look like).

Get Updates on the Splunk Community!

Now Playing: Splunk Education Summer Learning Premieres

It’s premiere season, and Splunk Education is rolling out new releases you won’t want to miss. Whether you’re ...

The Visibility Gap: Hybrid Networks and IT Services

The most forward thinking enterprises among us see their network as much more than infrastructure – it's their ...

Get Operational Insights Quickly with Natural Language on the Splunk Platform

In today’s fast-paced digital world, turning data into actionable insights is essential for success. With ...