Getting Data In

How do I exclude stream events based on criteria before indexing?

lznger88_2
Path Finder

Hi All,

I require to exclude events when the 'dest_port=80'. I have gone through other similar examples and have come to the conclusion that it is more poor REGEX skills causing the issue.

The stream data (netflow v9) goes through a heavy forwarder (HF) prior to the IDX. Below is the props.conf and transforms.conf of the HF, as well as event log:

props.conf:

[stream:netflow]
TRANSFORMS-null= setnull

transforms.conf:

[setnull]
REGEX = "dest_port":53
DEST_KEY = queue
FORMAT = nullQueue

I have also tried other REGEX, such as "(\w+)":80 , "([^\"]+)\":80 but they dont seem to be working.

Event log:

{"dest_ip":"123.456.789.99","dest_port":80,"event_name":"netFlowData","exporter_ip":"192.192.192.192","exporter_time":"2018-Jan-08 03:47:21","exporter_uptime":1269091254,"firewall_event":1,"flow_id":2950442453,"flow_start_time_milli":1546919225527,"input_snmpidx":5,"netflow_elements":["UNKNOWN : 0000","UNKNOWN : fa14fc27d49102c0abdf8bd4","UNKNOWN : 000000000000000000000000","UNKNOWN : 

Any help would be great. Thanks in advance

0 Karma
1 Solution

harsmarvania57
SplunkTrust
SplunkTrust

Hi,

Can you please try below config on HF

transforms.conf

[setnull]
REGEX = \"dest_port\"\:80
DEST_KEY = queue
FORMAT = nullQueue

View solution in original post

harsmarvania57
SplunkTrust
SplunkTrust

Hi,

Can you please try below config on HF

transforms.conf

[setnull]
REGEX = \"dest_port\"\:80
DEST_KEY = queue
FORMAT = nullQueue

lznger88_2
Path Finder

Thanks immensely harsmarvania57.

The regex you supplied outputs any dest_port when equal to 80* (for example, 80, 801, 8001, etc.) - tested this in Splunk.

I had to amend it slightly to get the right output based on the above log format:
REGEX = \"dest_port\":80\D

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Or you can use REGEX = \"dest_port\"\:80\,

0 Karma

lznger88_2
Path Finder

This didnt actually work given the above log format

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Using the Splunk Threat Research Team’s Latest Security Content

REGISTER HERE Tech Talk | Security Edition Did you know the Splunk Threat Research Team regularly releases ...

SplunkTrust | 2024 SplunkTrust Application Period is Open!

It's that time again, folks! That's right, the application/nomination period for the 2024 SplunkTrust is ...