Splunk Search

Discard unwanted syslog events?

mileserickson
Engager

I need to reduce our licensing usage by filtering common, valid, no-news-is-good-news domains out of our Barracuda Web Filter logs. I am trying to do this by sending such log messages to the nullQueue, but I clearly am not doing it correctly.

The set_bwf transform works, but the bwf_discard transform does not.

Given that I am a regex newbie, I suspect that I have crafted the regular expression under "[bwf_discard]" incorrectly.

Here's what I have so far:

props.conf:

[source::udp:514]
TRANSFORMS-discard=bwf_discard
TRANSFORMS-changesourcetype=set_bwf

[syslog:bwf]
REPORT-extract=bwf_extract,user

transforms.conf:

[bwf_discard]
REGEX = (?m)(commondomain1.com|commondomain2.com|commondomain3.com|commondomain4.com|commondomain5.com)
DEST_KEY = queue
FORMAT = nullQueue

[set_bwf]
REGEX = barracuda-hostname.localdomain
FORMAT = sourcetype::syslog:bwf
DEST_KEY = MetaData:Sourcetype

[bwf_extract]
DELIMS = " " 
FIELDS = bwf_month, bwf_day, bwf_time, bwf_hostname, bwf_daemon_info, bwf_timestamp, bwf_number_1, bwf_src_ip, bwf_dest_ip, bwf_content_type, bwf_src_ip2, bwf_dest_url, bwf_data_size, bwf_md5_anchor, bwf_action, bwf_reason, bwf_format_version, bwf_match_flag, bwf_TQ_flag, bwf_action_type, bwf_src_type, bwf_src_detail, bwf_dest_type, bwf_dest_detail, bwf_spy_type, bwf_spy_id, bwf_infection_score, bwf_matched_part, bwf_matched_category, bwf_user

[user]
REGEX = ([^\s]+)\s\[([\w\:]+)\]\s+$
FORMAT = category::$1 user::$2
Tags (3)

fineman
New Member

just have a look thomas sabo

0 Karma

Lowell
Super Champion

It looks like your regex should work, you don't need the (?m) for multi-line matching, and I would suggest making the regex slight more specific to the host field (rather than matching anywhere in the event), but that shouldn't stop it from working.

You can try this, but I suspect your issue is elsewhere.

[bwf_discard]
REGEX = ^[A-Z][a-z]{2} .\d \d\d:\d\d:\d\d (commondomain1\.com|commondomain2\.com|commondomain3\.com|commondomain4\.com|commondomain5\.com)\s
DEST_KEY = queue
FORMAT = nullQueue

If you don't have a regex testing tool, I'd highly suggested getting familiar with one. There are a number of free options out there, and it will save you time in the long run.

Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...