How can I avoid having lines that are commented within my files from being indexed by Splunk?
Lets say I have a log file that includes the following:
# Welcome to my file
# The lines with a # should be ignored by Splunk
12:34:56.123 Begin log file
12:34:58.123 Another event
How can I filter out the first two lines to prevent them from being indexed?
This can be done by adding the appropriate lines to your props.conf and transforms.conf files in order to make sure that only uncommented lines are indexed.
Given the above example, here is what you would want to have...
props.conf
[iis]
TRANSFORMS-noComments = ignoreComments
transforms.conf
[ignoreComments]
REGEX = ^[^#.*$].*$
DEST_KEY = queue
FORMAT = indexQueue
This will then ignore any line beginning with the # character and index the rest of the file.
This means if you like you can choose to replace whatever the # in the above regex with whatever comment character your lines begin with. Just be sure to escape any regex syntax characters with a backslash ().
You can also use this same regex in your Splunk searches (rex or regex commands) such as:
...| regex _raw="^[^#.*$].*$"
Also, you may wish to take a look at the alternate method (for sending IIS commented logs to the nullQueue) listed here:
http://splunk-base.splunk.com/answers/9313/ignoring-comments-in-log-files
I had similar issue and nothing worked for me.
Below props and transforms worked for me in my scenario where it ignored the lines started with #:
[sourcetype]
TRANSFORMS-ignore_comments = setnull
the above dint work for me.
This is what works in our environment
props.conf
[sourcetypeName]
TRANSFORMS-ignoreCommentedLines = ignoreLinesWithHash
[ignoreLinesWithHash]
REGEX = ^(?:[#].*)
DEST_KEY = queue
FORMAT = nullQueue
This can be done by adding the appropriate lines to your props.conf and transforms.conf files in order to make sure that only uncommented lines are indexed.
Given the above example, here is what you would want to have...
props.conf
[iis]
TRANSFORMS-noComments = ignoreComments
transforms.conf
[ignoreComments]
REGEX = ^[^#.*$].*$
DEST_KEY = queue
FORMAT = indexQueue
This will then ignore any line beginning with the # character and index the rest of the file.
This means if you like you can choose to replace whatever the # in the above regex with whatever comment character your lines begin with. Just be sure to escape any regex syntax characters with a backslash ().
You can also use this same regex in your Splunk searches (rex or regex commands) such as:
...| regex _raw="^[^#.*$].*$"
Also, you may wish to take a look at the alternate method (for sending IIS commented logs to the nullQueue) listed here:
http://splunk-base.splunk.com/answers/9313/ignoring-comments-in-log-files
I downvoted this post because regex doesn't work and the wrong format is specified if you want to ignore events. it should go to nullqueue
You need to configure props.conf and transforms.conf.
In your props.conf:
[your_sourcetype]
TRANSFORMS-xxx_log = setnull
In your transforms.conf:
[setnull]
REGEX = ^#.*\
DEST_KEY = queue
FORMAT = nullQueue
You can also refer to following ansower.
http://splunk-base.splunk.com/answers/11617/route-unwanted-logs-to-a-null-queue
thank you for your kindness.