Getting Data In

props and transform file modification issue

roopeshetty
Path Finder

Hi Guys,

 

In Splunk a field by name “event_sub_type” has multiple values. We don’t want to ingest any logs into splunk whose field “event_sub_type” value is either “WAN Firewall” or “TLS” (as marked in attached screen shot) as these are huge unwanted logs.

 

Capture.JPG

 

Our search query is : index=cato sourcetype=cato_source

 

We tried multiple ways by editing the props.conf and transforms.conf to exclude these logs as below but none of them are successful to exclude those logs;

 

props.conf

[sourcetype::cato_source]

TRANSFORMS-filter_logs = cloudparsing

 

transforms.conf

[cloudparsing]

REGEX = \"event_sub_type\":\"(WAN Firewall|TLS)\"

DEST_KEY = queue

FORMAT = nullQueue

 

Can someone please guide how to exclude these events whose “event_sub_type” value contains either “WAN Firewall” or “TLS” by editing props.conf and transforms.conf?

 

 

RAW Events for reference which needs to be excluded ;

1. event_sub_type":"WAN

 

{"event_count":1,"ISP_name":"Shanghai internet","rule":"Initial Connectivity Rule","dest_is_site_or_vpn":"Site","src_isp_ip":"0.0.0.0","time_str":"2023-11-28T04:27:40Z","src_site":"CHINA-AZURE-E2","src_ip":"0.0.0.1","internalId":"54464646","dest_site_name":"china_112,"event_type":"Security","src_country_code":"CN","action":"Monitor","subnet_name":"cn-001.net-vnet-1","pop_name":"Shanghai_1","dest_port":443,"dest_site":"china_connect","rule_name":"Initial Connectivity Rule","event_sub_type":"WAN Firewall","insertionDate":1701188916690,"ip_protocol":"TCP","rule_id":"101238","src_is_site_or_vpn":"Site","account_id":5555,"application":"HTTP(S)","src_site_name":"china_connect","src_country":"China","dest_ip":"0.0.0.0","os_type":"OS_ANDROID","app_stack""TCP","TLS","HTTP(S)"],"time":1701188860834}

 

2. "event_sub_type":"TLS","

 

{"event_count":4,"http_host_name":"isp.vpn","ISP_name":"China_internet","src_isp_ip":"0.0.0.0","tls_version":"TLSv1.3","time_str":"2023-11-28T04:27:16Z","src_site":"china_mtt","src_ip":"0.0.0.0","internalId":"rtrgrtr","domain_name":"china.gh.com","event_type":"Security","src_country_code":"CN","tls_error_description":"unknown CA","action":"Alert","subnet_name":"0.0.0.0/24","pop_name":"china_1","dest_port":443,"event_sub_type":"TLS","insertionDate":1701188915580,"dest_country_code":"SG","tls_error_type":"fatal","dns_name":"china.com","traffic_direction":"OUTBOUND","src_is_site_or_vpn":"Site","account_id":56565,"application":"Netskope","src_site_name":"CHINA-44","src_country":"China","dest_ip":"0.0.0.0","os_type":"OS_WINDOWS","time":1701188836011,"dest_country":"Singapore"}

 

 

Labels (2)
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi @roopeshetty ,

please try this regex in transforms.conf:

REGEX = \"event_sub_type\":\"(WAN|TLS)

Ciao.

Giuseppe

View solution in original post

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @roopeshetty ,

please try this regex in transforms.conf:

REGEX = \"event_sub_type\":\"(WAN|TLS)

Ciao.

Giuseppe

0 Karma

roopeshetty
Path Finder

Hi, Tried as below; still no luck , logs are keep coming; 

 

props.conf 

[sourcetype::cato_source]
TRANSFORMS-filter_logs = cloudparsing

 

transforms.conf

[cloudparsing]
REGEX = \"event_sub_type\":\"(WAN|TLS)
DEST_KEY = queue
FORMAT = nullQueue

 

 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @roopeshetty ,

where did you located props.conf and transforms.conf?

they must be located in the first full Splunk instance that the logs are passing through, in other words in the Indexers or (if present) in the intermediate Heavy Forwarder.

Ciao.

Giuseppe

0 Karma

roopeshetty
Path Finder

Hi

props.conf and transforms.conf are located on our splunk enterprise server on "splunk add on for AWS" app path; that is "D:\Program Files\Splunk\etc\apps\Splunk_TA_aws\local"

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @roopeshetty ,

yes, but where is the input for there data flow: in the same server or in a different Heavy Forwarder?

If in a different Heavy Forwarder, you have to put these props.conf and transforms.conf in it.

Ciao.

Giuseppe

0 Karma

roopeshetty
Path Finder

Hi,  input also located on the same server on same path;

Capture.JPG

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @roopeshetty ,

in the header of the props.conf, try to not use "sourcetype:

[cato_source]
TRANSFORMS-filter_logs = cloudparsing

Ciao.

Giuseppe

0 Karma

roopeshetty
Path Finder

Thanks a lot gcusello, 

It worked as expected. One last question , in below regex we are looking for the texts "WAN" and "TSL" 

REGEX = \"event_sub_type\":\"(WAN|TLS)

in case if we want to look for  texts "WAN Firewall" and "TSL" , how the regex would be?

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @roopeshetty ,

you can use this regex:

REGEX = \"event_sub_type\":\"((WAN\s+Firewall)|TLS)

that you can test at https://regex101.com/r/YBCWAB/1

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉

0 Karma

roopeshetty
Path Finder

thanks a lot.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @roopeshetty ,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉

0 Karma
Get Updates on the Splunk Community!

Database Performance Sidebar Panel Now on APM Database Query Performance & Service ...

We’ve streamlined the troubleshooting experience for database-related service issues by adding a database ...

IM Landing Page Filter - Now Available

We’ve added the capability for you to filter across the summary details on the main Infrastructure Monitoring ...

Dynamic Links from Alerts to IM Navigators - New in Observability Cloud

Splunk continues to improve the troubleshooting experience in Observability Cloud with this latest enhancement ...