props and transform file modification issue

[roopeshetty]

Hi Guys,

In Splunk a field by name “event_sub_type” has multiple values. We don’t want to ingest any logs into splunk whose field “event_sub_type” value is either “WAN Firewall” or “TLS” (as marked in attached screen shot) as these are huge unwanted logs.

Our search query is : index=cato sourcetype=cato_source

We tried multiple ways by editing the props.conf and transforms.conf to exclude these logs as below but none of them are successful to exclude those logs;

props.conf

[sourcetype::cato_source]

TRANSFORMS-filter_logs = cloudparsing

transforms.conf

[cloudparsing]

REGEX = \"event_sub_type\":\"(WAN Firewall|TLS)\"

DEST_KEY = queue

FORMAT = nullQueue

Can someone please guide how to exclude these events whose “event_sub_type” value contains either “WAN Firewall” or “TLS” by editing props.conf and transforms.conf?

RAW Events for reference which needs to be excluded ;

1. event_sub_type":"WAN

{"event_count":1,"ISP_name":"Shanghai internet","rule":"Initial Connectivity Rule","dest_is_site_or_vpn":"Site","src_isp_ip":"0.0.0.0","time_str":"2023-11-28T04:27:40Z","src_site":"CHINA-AZURE-E2","src_ip":"0.0.0.1","internalId":"54464646","dest_site_name":"china_112,"event_type":"Security","src_country_code":"CN","action":"Monitor","subnet_name":"cn-001.net-vnet-1","pop_name":"Shanghai_1","dest_port":443,"dest_site":"china_connect","rule_name":"Initial Connectivity Rule","event_sub_type":"WAN Firewall","insertionDate":1701188916690,"ip_protocol":"TCP","rule_id":"101238","src_is_site_or_vpn":"Site","account_id":5555,"application":"HTTP(S)","src_site_name":"china_connect","src_country":"China","dest_ip":"0.0.0.0","os_type":"OS_ANDROID","app_stack""TCP","TLS","HTTP(S)"],"time":1701188860834}

2. "event_sub_type":"TLS","

{"event_count":4,"http_host_name":"isp.vpn","ISP_name":"China_internet","src_isp_ip":"0.0.0.0","tls_version":"TLSv1.3","time_str":"2023-11-28T04:27:16Z","src_site":"china_mtt","src_ip":"0.0.0.0","internalId":"rtrgrtr","domain_name":"china.gh.com","event_type":"Security","src_country_code":"CN","tls_error_description":"unknown CA","action":"Alert","subnet_name":"0.0.0.0/24","pop_name":"china_1","dest_port":443,"event_sub_type":"TLS","insertionDate":1701188915580,"dest_country_code":"SG","tls_error_type":"fatal","dns_name":"china.com","traffic_direction":"OUTBOUND","src_is_site_or_vpn":"Site","account_id":56565,"application":"Netskope","src_site_name":"CHINA-44","src_country":"China","dest_ip":"0.0.0.0","os_type":"OS_WINDOWS","time":1701188836011,"dest_country":"Singapore"}

[gcusello · Accepted solution]

Hi @roopeshetty ,

please try this regex in transforms.conf:

REGEX = \"event_sub_type\":\"(WAN|TLS)

Ciao.

Giuseppe

[roopeshetty]

Hi, Tried as below; still no luck , logs are keep coming; 

props.conf 

[sourcetype::cato_source]
TRANSFORMS-filter_logs = cloudparsing

transforms.conf

[cloudparsing]
REGEX = \"event_sub_type\":\"(WAN|TLS)
DEST_KEY = queue
FORMAT = nullQueue

[gcusello]

Hi @roopeshetty ,

where did you located props.conf and transforms.conf?

they must be located in the first full Splunk instance that the logs are passing through, in other words in the Indexers or (if present) in the intermediate Heavy Forwarder.

Ciao.

Giuseppe

[roopeshetty]

Hi

props.conf and transforms.conf are located on our splunk enterprise server on "splunk add on for AWS" app path; that is "D:\Program Files\Splunk\etc\apps\Splunk_TA_aws\local"

[gcusello]

Hi @roopeshetty ,

yes, but where is the input for there data flow: in the same server or in a different Heavy Forwarder?

If in a different Heavy Forwarder, you have to put these props.conf and transforms.conf in it.

Ciao.

Giuseppe

[roopeshetty]

Hi,  input also located on the same server on same path;

[gcusello]

Hi @roopeshetty ,

in the header of the props.conf, try to not use "sourcetype:

[cato_source]
TRANSFORMS-filter_logs = cloudparsing

Ciao.

Giuseppe

[roopeshetty]

Thanks a lot gcusello, 

It worked as expected. One last question , in below regex we are looking for the texts "WAN" and "TSL" 

REGEX = \"event_sub_type\":\"(WAN|TLS)

in case if we want to look for  texts "WAN Firewall" and "TSL" , how the regex would be?

[gcusello]

Hi @roopeshetty ,

you can use this regex:

REGEX = \"event_sub_type\":\"((WAN\s+Firewall)|TLS)

that you can test at https://regex101.com/r/YBCWAB/1

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉

[roopeshetty]

thanks a lot.

[gcusello]

Hi @roopeshetty ,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉