topic Re: do not index events using props/transforms regex help in Getting Data In

do not index events using props/transforms regex help

mjm295 — Fri, 12 Jun 2020 02:31:03 GMT

A recent agent install across our infrastructure has created a flood in the proxy logs of blocked messages which is blowing out our license.

Until there is a proper fix I need to stop ingesting events related to 1 URL.

I think I just need help with the REGEX part.

Log are dropped to a syslog server running a heavy forwarder, then we run a monitor on that log file.

Inputs.conf

[monitor:///remotesyslogs/mgmt-austaiaecho00*/*.log] disabled = false index = star_proxy sourcetype = cisco:wsa:squid

props.conf

[cisco:wsa:squid] TRANSFORMS-screen=eliminate-screenconnect

transforms.conf

[eliminate-screenconnect] REGEX = ?.=screenconnect DEST_KEYi = queue FORMAT = nullQueue

Example event from log file:

2020-06-12T04:04:55+10:00 mgmt-austaiaecho005.casino.internal accesslogs_splunk: Info: 1591898695.320 0 10.10.216.100 TCP_DENIED/407 0 CONNECT tunnel://screenconnect.techmedia.com.au:8080/ - NONE/- - OTHER-NONE-DefaultGroup-NONE-NONE-NONE-NONE <-,-,-,"-",-,-,-,-,"-",-,-,-,"-",-,-,"-","-",-,-,-,-,"-","-","-","-","-","-",0.00,0,-,"-","-",-,"-",-,-,"-","-",-,-,"-"> - - "12/Jun/2020:04:04:55 +1000" -

Basically for next 4 weeks I need to drop all events with

screenconnect.techmedia.com.au

Thanks

Mark

Re: do not index events using props/transforms regex help

alonsocaio — Sat, 20 Jun 2020 14:25:06 GMT

Hi,

I would try to use some regex like this one:

^.+screenconnect\.techmedia\.com\.au.+$

I have tested it using Regex101 (https://regex101.com/r/gHJVDp/2), with the URLs below:

screenconnect.techmedia.com.au screenconnect2.techmedia.com.au screenconnect.techmedia10.com.au

* I also recommend you to test this regex (and any other you build or find) in a dev environment, before using it in your production transforms.conf file.

Re: do not index events using props/transforms regex help

to4kawa — Sun, 21 Jun 2020 01:02:18 GMT

[eliminate-screenconnect] REGEX = screenconnect\.techmedia\.com\.au DEST_KEY = queue FORMAT = nullQueue

A simple REGEX is enough for nullQueue.