topic Re: How to filter out lines that start with # in Getting Data In

How to filter out lines that start with #

mlevsh — Wed, 17 Nov 2021 21:08:16 GMT

We have logs , where first few lines start with "#" and we don't need to ingest these lines.

We tired to use different methods , that didn't work. Appreciated the help/ideas from splunkers:

1st idea: use PREAMBLE_REGEX = ^#.* in props.conf on Heavy Forwarders where data are being parsed

2nd idea : use TRANSFORMS-null = setnull in props.conf and transforms.conf
on Heavy Forwarders where data are being parsed

transforms.conf:
[setnull]
REGEX = ^#.*
DEST_KEY = queue
FORMAT = nullQueue

example of log:

#-----------------------------------------
#DATE CREATED: 11/02/2021@04:16
#SUBJECT: REPORT ON THE GENERAL STATUS OF AUTOSYS JOBS
#ENVIRONMENT: CBA
#-----------------------------------------

11/02/2021@04:16,CBA,OTHER,CBA_CLIENT_REPORT_BOX,OI
11/02/2021@04:16,CBA,OTHER,CBA_copy_file_job,OI
11/02/2021@04:16,CBA,OTHER,CBA_ABC_SCHEDULER_BOX,OI
11/02/2021@04:16,CBA,OTHER,CBA_ABC_REPORT_BOX,OI

Re: How to filter out lines that start with #

PickleRick — Wed, 17 Nov 2021 21:29:21 GMT

Hash is a character used for comments both in your logs and splunk config files. You might try escaping it.

Re: How to filter out lines that start with #

mlevsh — Wed, 17 Nov 2021 23:54:45 GMT

Hi PickleRick,

Thank you for suggestion. Unfortunately , it didn't work

Re: How to filter out lines that start with #

bhargavi — Thu, 18 Nov 2021 05:42:00 GMT

Hi @mlevsh

Try this for preamble_regex. It works.

PREAMBLE_REGEX=#

If this helps, give thumbs-up 🙂

Happy Splunking!!