topic How do I send events to nullqueue? in Getting Data In

How do I send events to nullqueue?

stevenbutterwor — Fri, 30 Aug 2019 09:42:39 GMT

I am currently ingesting AWS VPC Flow logs from our AWS tenant. Most of the logs are internal traffic between ec2 instances. I'd like to send these events to nullqueue as they are not much use to us, we are only concerned with inbound and outbound traffic to the VPC. I think the best way to drop these events is to insert some regex into transforms.conf? Could someone help with this?

I wish to exclude events that have both srcaddr starting with 10.x and dstaddr starting with 10.x

Any help would be appreciated

Re: How do I send events to nullqueue?

sandyIscream — Fri, 30 Aug 2019 09:58:54 GMT

@stevenbutterworth You have to setup two files. props.conf and transforms.conf

Please have a loot at the example below.

props.conf
[srctype1]
TRANSFORMS-set = setnull, logstoCapture

transforms.conf
[logstoCapture]
REGEX =
DEST_KEY = queue
FORMAT = indexQueue

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

Re: How do I send events to nullqueue?

stevenbutterwor — Fri, 30 Aug 2019 10:09:15 GMT

Thanks for your answer - it's regex part I need help with

Re: How do I send events to nullqueue?

sandyIscream — Fri, 30 Aug 2019 10:10:44 GMT

Could you please send the sample log file. Then I might be able to help you write the regex query

Re: How do I send events to nullqueue?

stevenbutterwor — Fri, 30 Aug 2019 12:06:08 GMT

I have attached an image. The 2 field names are srcaddr and dstaddr. It's when both of these start with 10 I want to send to nullQueue

Re: How do I send events to nullqueue?

jawaharas — Fri, 30 Aug 2019 13:28:44 GMT

You can try below RegEx.

[setnull]
REGEX = ^\d\s\d*\seni-\d{2}\s10\.\d{1,3}\.\d{1,3}\.\d{1,3}\s10\.\d{1,3}\.\d{1,3}\.\d{1,3}
DEST_KEY = queue
FORMAT = nullQueue

Re: How do I send events to nullqueue?

mayurr98 — Fri, 30 Aug 2019 13:58:46 GMT

try this regex:

eni-\d{8}\s+10\d\.\d{1,3}\.\d{1,3}.\.\d{1,3}\s+10\d\.\d{1,3}\.\d{1,3}\.\d{1,3}

Re: How do I send events to nullqueue?

sandyIscream — Tue, 03 Sep 2019 02:16:10 GMT

@stevenbutterworth Did you try the regex that people have posted here ? If that helps then could you please accept the answer.

Re: How do I send events to nullqueue?

stevenbutterwor — Wed, 30 Sep 2020 02:03:17 GMT

The correct regex is as follows

^\d\s\d*\seni-.*\s10.\d{1,3}.\d{1,3}.\d{1,3}\s10.\d{1,3}.\d{1,3}.\d{1,3}