topic Re: Filtering Specific Data from a host in Splunk Search

How to filter Specific Data from a host?

nick_currie — Wed, 11 May 2022 22:53:14 GMT

Hi there - I am trying to filter out some noisy rules in a specific firewall (FWCL01) from being ingested into splunk.

On my Heavy forwearder that send into splunk i have applied the following props.conf and transform.conf

PROPS.CONF

[host::FWCL01]
TRANSFORMS-set_null = FWCL01_ruleid0_to_null, FWCL01_ruleid4_to_null

TRANSFORMS.CONF

[FWCL01_ruleid0_to_null]
REGEX = policyid=0
DEST_KEY = queue
FORMAT = nullQueue

[FWCL01_ruleid4_to_null]
REGEX = policyid=4
DEST_KEY = queue
FORMAT = nullQueue

This doesnt seem to work. However when i change props.conf to us the sourcetype [fgt-traffic] as per below it works

[fgt_traffic]

TRANSFORMS-set_null = FWCL01_ruleid0_to_null, FWCL01_ruleid4_to_null

The logs show as following:

May 11 16:12:54 10.8.11.1 logver=602101263 timestamp=1652256773 devname="FWCL01" devid="XXXXXXX" vd="Outer-DMZ" date=2022-05-11 time=16:12:53 logid="0000000013" type="traffic" subtype="forward" level="notice" eventtime=1652256774280610010 tz="+0800" srcip=45.143.203.10 srcport=8080 srcintf="XXXX" srcintfrole="lan" dstip=XXXX dstport=8088 dstintf="XXXX" dstintfrole="undefined" srcinetsvc="Malicious-Malicious.Server" sessionid=2932531463 proto=6 action="deny" policyid=4 policytype="policy" poluuid="XXXXX" service="tcp/8088" dstcountry="Australia" srccountry="Netherlands" trandisp="noop" duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 appcat="unscanned" crscore=30 craction=131072 crlevel="high" mastersrcmac="XXXXX" srcmac="XXXXX" srcserver=0

When i use btool it looks like the correct props are being applied

D:\Program Files\Splunk\bin>splunk btool props list | findstr FWCL01
[host::FWCL01]
TRANSFORMS-set_null = FWCL01_ruleid0_to_null, FWCL01_ruleid4_to_null

Any idea's?

Re: Filtering Specific Data from a host

PickleRick — Wed, 11 May 2022 08:36:45 GMT

At first glance looks pretty OK. Are you 100% sure the host value is right? (bonus question - isn't the host value extracted and overwritten in transforms?)

Re: Filtering Specific Data from a host

nick_currie — Wed, 11 May 2022 08:45:14 GMT

Aha - OK this might be where I am going wrong. The host is right - but I cant see the host field within the event log entry when i look at the source.. Is this why its not triggering? do I need to use devname field devname="FWCL01"?

These logs are sent from a Fortianalyzer to a syslog - so perhaps the Host value is generated in a different part of the process

Re: Filtering Specific Data from a host

PickleRick — Wed, 11 May 2022 09:02:30 GMT

The question is how you're getting that data. Typically the host is either set for a specific input, or might be (for example with HEC) pushed by the source with the event data.

Re: Filtering Specific Data from a host

nick_currie — Wed, 11 May 2022 09:09:16 GMT

Sorry bear with me here - i have inherited this environment and am a splunk n00b -

So it looks like we have the Splunk_TA_fortinet_fortigate app installed and this generates the hostname from the devname based on the transforms.conf file in that app as shown below: does this mean i cannot filter on HF's based on the host value?

##sourcetype
[force_sourcetype_fgt]
SOURCE_KEY = _raw
DEST_KEY = MetaData:Sourcetype
REGEX = ^.+?devid=\"?F(?:G|W|6K).+?(?:\s|\,|\,\s)type=\"?(traffic|utm|event|anomaly)
FORMAT = sourcetype::fgt_$1

[fgt_change_hostname]
SOURCE_KEY = _raw
DEST_KEY = MetaData:Host
REGEX = ^.+?devname=\"(\S+)\"\s
FORMAT = host::$1

## LOOKUP

[ftnt_protocol_lookup]
filename = ftnt_protocol_info.csv

[ftnt_action_lookup]
filename = ftnt_action_info.csv

[ftnt_event_action_lookup]
filename = ftnt_event_action_info.csv

## REPORT

[field_extract]
DELIMS = "\ ,", "="

Re: Filtering Specific Data from a host

PickleRick — Wed, 11 May 2022 09:34:14 GMT

Apparently the priorities are so that [source::*] pattern settings are applied first, then [host::*] and at the end the general sourcetype settings. And the resulting settings to be applied are decided as far as I remember with the values at the beginning of the parsing/transforming process (so that overwritten field values are not taken into account), you can'd match to this value of yours. (as a side trivia - you cannot make a loop with overwriting metadata; I tried ;-)). So you have to either attach your transforms to the sourcetype-level settings or check for the original host field value, before rewriting. It will most probably be either set on the input or will come from the hostname of the forwarder getting the events from your fortigate devices.

Re: Filtering Specific Data from a host

nick_currie — Wed, 11 May 2022 11:41:22 GMT

Thank you very much for all your help Rick! Unfortunately the original host is a syslog server that has a few different input files - however the file that holds all the forti events is a single input as it's aggregated byour Fortianalyzer device.

Plan B i think will have to be a fairly lengthy regexp that has both the policy ID and deviceid. Our Heavy Forwarders have resonable processing power however they are already sitting around 50% util - hopefully this extra pattern matching will not create too much of an overhead.