Splunk Search

How can I not index commented lines from my files?

Rob
Splunk Employee
Splunk Employee

How can I avoid having lines that are commented within my files from being indexed by Splunk?

Lets say I have a log file that includes the following:

# Welcome to my file
# The lines with a # should be ignored by Splunk
12:34:56.123 Begin log file
12:34:58.123 Another event

How can I filter out the first two lines to prevent them from being indexed?

1 Solution

Rob
Splunk Employee
Splunk Employee

This can be done by adding the appropriate lines to your props.conf and transforms.conf files in order to make sure that only uncommented lines are indexed.

Given the above example, here is what you would want to have...

props.conf

[iis]
TRANSFORMS-noComments = ignoreComments

transforms.conf

[ignoreComments]
REGEX = ^[^#.*$].*$
DEST_KEY = queue
FORMAT = indexQueue

This will then ignore any line beginning with the # character and index the rest of the file.

This means if you like you can choose to replace whatever the # in the above regex with whatever comment character your lines begin with. Just be sure to escape any regex syntax characters with a backslash ().

You can also use this same regex in your Splunk searches (rex or regex commands) such as:

...| regex _raw="^[^#.*$].*$"

Also, you may wish to take a look at the alternate method (for sending IIS commented logs to the nullQueue) listed here:
http://splunk-base.splunk.com/answers/9313/ignoring-comments-in-log-files

View solution in original post

nithish
Loves-to-Learn

I had similar issue and nothing worked for me.

Below props and transforms worked for me in my scenario where it ignored the lines started with #:

transforms.conf:


[setnull]
REGEX=^[#\r\n]
DEST_KEY=queue
FORMAT=nullQueue

props.conf:

[sourcetype]
TRANSFORMS-ignore_comments = setnull

0 Karma

deepamshah
Explorer

the above dint work for me.

This is what works in our environment

props.conf
[sourcetypeName]
TRANSFORMS-ignoreCommentedLines = ignoreLinesWithHash

[ignoreLinesWithHash]
REGEX = ^(?:[#].*)
DEST_KEY = queue
FORMAT = nullQueue

Rob
Splunk Employee
Splunk Employee

This can be done by adding the appropriate lines to your props.conf and transforms.conf files in order to make sure that only uncommented lines are indexed.

Given the above example, here is what you would want to have...

props.conf

[iis]
TRANSFORMS-noComments = ignoreComments

transforms.conf

[ignoreComments]
REGEX = ^[^#.*$].*$
DEST_KEY = queue
FORMAT = indexQueue

This will then ignore any line beginning with the # character and index the rest of the file.

This means if you like you can choose to replace whatever the # in the above regex with whatever comment character your lines begin with. Just be sure to escape any regex syntax characters with a backslash ().

You can also use this same regex in your Splunk searches (rex or regex commands) such as:

...| regex _raw="^[^#.*$].*$"

Also, you may wish to take a look at the alternate method (for sending IIS commented logs to the nullQueue) listed here:
http://splunk-base.splunk.com/answers/9313/ignoring-comments-in-log-files

nealpotter
Engager

I downvoted this post because regex doesn't work and the wrong format is specified if you want to ignore events. it should go to nullqueue

0 Karma

Takajian
Builder

You need to configure props.conf and transforms.conf.

In your props.conf:

[your_sourcetype]
TRANSFORMS-xxx_log = setnull

In your transforms.conf:

[setnull]
REGEX = ^#.*\
DEST_KEY = queue
FORMAT = nullQueue

You can also refer to following ansower.

http://splunk-base.splunk.com/answers/11617/route-unwanted-logs-to-a-null-queue

kjycls
Engager

thank you for your kindness.

0 Karma
Get Updates on the Splunk Community!

Your Guide to SPL2 at .conf24!

So, you’re headed to .conf24? You’re in for a good time. Las Vegas weather is just *chef’s kiss* beautiful in ...

Get ready to show some Splunk Certification swagger at .conf24!

Dive into the deep end of data by earning a Splunk Certification at .conf24. We're enticing you again this ...

Built-in Service Level Objectives Management to Bridge the Gap Between Service & ...

Now On-Demand Join us to learn more about how you can leverage Service Level Objectives (SLOs) and the new ...