How do I exclude stream events based on criteria before indexing?

[lznger88_2]

Hi All,

I require to exclude events when the 'dest_port=80'. I have gone through other similar examples and have come to the conclusion that it is more poor REGEX skills causing the issue.

The stream data (netflow v9) goes through a heavy forwarder (HF) prior to the IDX. Below is the props.conf and transforms.conf of the HF, as well as event log:

props.conf:

[stream:netflow]
TRANSFORMS-null= setnull

transforms.conf:

[setnull]
REGEX = "dest_port":53
DEST_KEY = queue
FORMAT = nullQueue

I have also tried other REGEX, such as "(\w+)":80 , "([^\"]+)\":80 but they dont seem to be working.

Event log:

{"dest_ip":"123.456.789.99","dest_port":80,"event_name":"netFlowData","exporter_ip":"192.192.192.192","exporter_time":"2018-Jan-08 03:47:21","exporter_uptime":1269091254,"firewall_event":1,"flow_id":2950442453,"flow_start_time_milli":1546919225527,"input_snmpidx":5,"netflow_elements":["UNKNOWN : 0000","UNKNOWN : fa14fc27d49102c0abdf8bd4","UNKNOWN : 000000000000000000000000","UNKNOWN : 

Any help would be great. Thanks in advance

[harsmarvania57 · Accepted solution]

Hi,

Can you please try below config on HF

transforms.conf

[setnull]
REGEX = \"dest_port\"\:80
DEST_KEY = queue
FORMAT = nullQueue

[lznger88_2]

Thanks immensely harsmarvania57.

The regex you supplied outputs any dest_port when equal to 80* (for example, 80, 801, 8001, etc.) - tested this in Splunk.

I had to amend it slightly to get the right output based on the above log format:
REGEX = \"dest_port\":80\D

[harsmarvania57]

Or you can use REGEX = \"dest_port\"\:80\,

[lznger88_2]

This didnt actually work given the above log format