topic Re: Filtering data out with RegEx based on blacklist works only partially in Getting Data In

Filtering data out with RegEx based on blacklist works only partially

kepffr — Mon, 25 Jan 2021 14:36:07 GMT

Hi guys!

I want to filter data out on my forwarder by using Regular Expression in transforms.conf. The strange thing is, that it only works partially but my regex itself is or should be fine.

transforms.conf

props.conf

[STANZA_NAME] TRANSFORMS-DeleteStuff = deleteAdvertisingTracking,deleteShopping

The second Stanza named "deleteShopping" works just fine but not the first. I could observe that it stopped working with 3 or more substrings (e.g adnxs|doubleclick|adsafeprotected|pubmatic) in the regex. I've tried adding "LOOKAHEAD = 65535" but that didn't help.

Of course I restarted the forwarder after the changes. Do you have any idea what's going wrong? I'm using Splunk v8.1.1.

Re: Filtering data out with RegEx based on blacklist works only partially

kepffr — Wed, 27 Jan 2021 16:38:03 GMT

Hi again. I've tried more things but nothing worked. I think my assumption that it works with less substrings is wrong, I cannot reproduce that anymore. Things I've tried:

- Set MATCH_LIMIT to a higher value

- Replaced the sourcetype in the stanza specified in props.conf with source::tcp:1422

- Included a named group in the regex, e.g:

REGEX=(\t)hostname=(.*\.)?(?<site>amazon|ebay)\.(de|com|net)

Data to test:

2021-01-27 16:55:42 action=Allowed event_id=6922469125184356358 protocol=SSL category=Corporate Marketing dest=1.1.1.1 http_referrer=None http_user_agent=XXXXXXX clientpublicIP=1.1.1 status=NA user=something.something@something.com url=ebayimg.ebay.com hostname=ebayimg.ebay.com clientIP=1.1.1 threatcategory=None threatname=None appname=XXX pagerisk=0 department=XXXX supercategory=XXXX appclass=File Share urlclass=XXXXX threatclass=None bytes_out=2272 bytes_in=5100

props.conf

[XXXXXX] DATETIME_CONFIG = CURRENT TZ = UTC [source::tcp:1422] TRANSFORMS-DeleteStuff = deleteAdvertisingTracking,deleteShopping

transforms.conf

This is what the command "cmd btool transforms list deleteAds" says:

CAN_OPTIMIZE = True
CLEAN_KEYS = True
DEFAULT_VALUE =
DEPTH_LIMIT = 1000
DEST_KEY = queue
FORMAT = nullQueue
KEEP_EMPTY_VALS = False
LOOKAHEAD = 4096
MATCH_LIMIT = 100000
MV_ADD = False
REGEX = (\t)hostname=(.*\.)?(amazon|ebay)\.(de|com|net|cn)
SOURCE_KEY = _raw
WRITE_META = False

Re: Filtering data out with RegEx based on blacklist works only partially

manjunathmeti — Wed, 27 Jan 2021 17:56:53 GMT

Is stanza name deleteAdvertisingTracking OR deleteAds in transformas.conf?

Re: Filtering data out with RegEx based on blacklist works only partially

kepffr — Thu, 28 Jan 2021 08:26:37 GMT

Both, I want to execute both stanzas. Is that not possible?

Re: Filtering data out with RegEx based on blacklist works only partially

manjunathmeti — Thu, 28 Jan 2021 12:25:57 GMT

Yes, it is possible. Make sure that regex of any one of them is matching to the logs you want to send to the null queue.

Re: Filtering data out with RegEx based on blacklist works only partially

kepffr — Thu, 28 Jan 2021 14:32:35 GMT

Hi thanks for your response. The regex matches it, that's my problem.

Re: Filtering data out with RegEx based on blacklist works only partially

manjunathmeti — Fri, 29 Jan 2021 04:01:35 GMT

Try with the below configurations without "\".