topic Re: How do I exclude stream events based on criteria before indexing? in Getting Data In

How do I exclude stream events based on criteria before indexing?

lznger88_2 — Tue, 08 Jan 2019 05:01:28 GMT

Hi All,

I require to exclude events when the 'dest_port=80'. I have gone through other similar examples and have come to the conclusion that it is more poor REGEX skills causing the issue.

The stream data (netflow v9) goes through a heavy forwarder (HF) prior to the IDX. Below is the props.conf and transforms.conf of the HF, as well as event log:

props.conf:

[stream:netflow]
TRANSFORMS-null= setnull

transforms.conf:

[setnull]
REGEX = "dest_port":53
DEST_KEY = queue
FORMAT = nullQueue

I have also tried other REGEX, such as "(\w+)":80 , "([^\"]+)\":80 but they dont seem to be working.

Event log:

{"dest_ip":"123.456.789.99","dest_port":80,"event_name":"netFlowData","exporter_ip":"192.192.192.192","exporter_time":"2018-Jan-08 03:47:21","exporter_uptime":1269091254,"firewall_event":1,"flow_id":2950442453,"flow_start_time_milli":1546919225527,"input_snmpidx":5,"netflow_elements":["UNKNOWN : 0000","UNKNOWN : fa14fc27d49102c0abdf8bd4","UNKNOWN : 000000000000000000000000","UNKNOWN :

Any help would be great. Thanks in advance

Re: How do I exclude stream events based on criteria before indexing?

harsmarvania57 — Tue, 08 Jan 2019 13:30:29 GMT

Hi,

Can you please try below config on HF

transforms.conf

[setnull]
REGEX = \"dest_port\"\:80
DEST_KEY = queue
FORMAT = nullQueue

Re: How do I exclude stream events based on criteria before indexing?

lznger88_2 — Wed, 09 Jan 2019 03:57:49 GMT

Thanks immensely harsmarvania57.

The regex you supplied outputs any dest_port when equal to 80* (for example, 80, 801, 8001, etc.) - tested this in Splunk.

I had to amend it slightly to get the right output based on the above log format:
REGEX = \"dest_port\":80\D

Re: How do I exclude stream events based on criteria before indexing?

harsmarvania57 — Wed, 09 Jan 2019 08:56:32 GMT

Or you can use REGEX = \"dest_port\"\:80\,

Re: How do I exclude stream events based on criteria before indexing?

lznger88_2 — Fri, 18 Jan 2019 04:01:09 GMT

This didnt actually work given the above log format