How to filter out lines that start with #

[mlevsh]

We have logs , where first few lines start with "#" and we don't need to ingest these lines. 

We tired to use different methods , that didn't work. Appreciated the help/ideas from splunkers:

1st idea: use PREAMBLE_REGEX = ^#.* in props.conf  on Heavy Forwarders where data are being parsed

2nd idea : use TRANSFORMS-null = setnull in props.conf  and transforms.conf
on Heavy Forwarders where data are being parsed

transforms.conf:
[setnull]
REGEX = ^#.*
DEST_KEY = queue
FORMAT = nullQueue

example of log:

#-----------------------------------------
#DATE CREATED:  11/02/2021@04:16
#SUBJECT:       REPORT ON THE GENERAL STATUS OF AUTOSYS JOBS
#ENVIRONMENT:   CBA
#-----------------------------------------

11/02/2021@04:16,CBA,OTHER,CBA_CLIENT_REPORT_BOX,OI
11/02/2021@04:16,CBA,OTHER,CBA_copy_file_job,OI
11/02/2021@04:16,CBA,OTHER,CBA_ABC_SCHEDULER_BOX,OI
11/02/2021@04:16,CBA,OTHER,CBA_ABC_REPORT_BOX,OI

[bhargavi]

Hi @mlevsh 

Try this for preamble_regex. It works.

PREAMBLE_REGEX=#

bhargavi_0-1637214105272.png

If this helps, give thumbs-up 🙂

Happy Splunking!!

[PickleRick]

Hash is a character used for comments both in your logs and splunk config files. You might try escaping it.

[mlevsh]

Hi PickleRick,

Thank you for suggestion. Unfortunately , it didn't work