Discard unwanted syslog events?

[mileserickson]

I need to reduce our licensing usage by filtering common, valid, no-news-is-good-news domains out of our Barracuda Web Filter logs. I am trying to do this by sending such log messages to the nullQueue, but I clearly am not doing it correctly.

The set_bwf transform works, but the bwf_discard transform does not.

Given that I am a regex newbie, I suspect that I have crafted the regular expression under "[bwf_discard]" incorrectly.

Here's what I have so far:

props.conf:

[source::udp:514]
TRANSFORMS-discard=bwf_discard
TRANSFORMS-changesourcetype=set_bwf

[syslog:bwf]
REPORT-extract=bwf_extract,user

transforms.conf:

[bwf_discard]
REGEX = (?m)(commondomain1.com|commondomain2.com|commondomain3.com|commondomain4.com|commondomain5.com)
DEST_KEY = queue
FORMAT = nullQueue

[set_bwf]
REGEX = barracuda-hostname.localdomain
FORMAT = sourcetype::syslog:bwf
DEST_KEY = MetaData:Sourcetype

[bwf_extract]
DELIMS = " " 
FIELDS = bwf_month, bwf_day, bwf_time, bwf_hostname, bwf_daemon_info, bwf_timestamp, bwf_number_1, bwf_src_ip, bwf_dest_ip, bwf_content_type, bwf_src_ip2, bwf_dest_url, bwf_data_size, bwf_md5_anchor, bwf_action, bwf_reason, bwf_format_version, bwf_match_flag, bwf_TQ_flag, bwf_action_type, bwf_src_type, bwf_src_detail, bwf_dest_type, bwf_dest_detail, bwf_spy_type, bwf_spy_id, bwf_infection_score, bwf_matched_part, bwf_matched_category, bwf_user

[user]
REGEX = ([^\s]+)\s\[([\w\:]+)\]\s+$
FORMAT = category::$1 user::$2

[fineman]

just have a look thomas sabo

[Lowell]

It looks like your regex should work, you don't need the (?m) for multi-line matching, and I would suggest making the regex slight more specific to the host field (rather than matching anywhere in the event), but that shouldn't stop it from working.

You can try this, but I suspect your issue is elsewhere.

[bwf_discard]
REGEX = ^[A-Z][a-z]{2} .\d \d\d:\d\d:\d\d (commondomain1\.com|commondomain2\.com|commondomain3\.com|commondomain4\.com|commondomain5\.com)\s
DEST_KEY = queue
FORMAT = nullQueue

If you don't have a regex testing tool, I'd highly suggested getting familiar with one. There are a number of free options out there, and it will save you time in the long run.