Splunk Search

How to filter Specific Data from a host?

nick_currie
Path Finder

Hi there - I am trying to filter out some noisy rules in a specific firewall (FWCL01) from being ingested into splunk.

 

On my Heavy forwearder that send into splunk i have applied the following props.conf and transform.conf

 

PROPS.CONF

[host::FWCL01]
TRANSFORMS-set_null = FWCL01_ruleid0_to_null, FWCL01_ruleid4_to_null

 

TRANSFORMS.CONF

[FWCL01_ruleid0_to_null]
REGEX = policyid=0
DEST_KEY = queue
FORMAT = nullQueue

[FWCL01_ruleid4_to_null]
REGEX = policyid=4
DEST_KEY = queue
FORMAT = nullQueue

 

 

This doesnt seem to work. However when i change props.conf to us the sourcetype [fgt-traffic] as per below it works

[fgt_traffic]

TRANSFORMS-set_null = FWCL01_ruleid0_to_null, FWCL01_ruleid4_to_null

 

 

The logs show as following:

May 11 16:12:54 10.8.11.1 logver=602101263 timestamp=1652256773 devname="FWCL01" devid="XXXXXXX" vd="Outer-DMZ" date=2022-05-11 time=16:12:53 logid="0000000013" type="traffic" subtype="forward" level="notice" eventtime=1652256774280610010 tz="+0800" srcip=45.143.203.10 srcport=8080 srcintf="XXXX" srcintfrole="lan" dstip=XXXX dstport=8088 dstintf="XXXX" dstintfrole="undefined" srcinetsvc="Malicious-Malicious.Server" sessionid=2932531463 proto=6 action="deny" policyid=4 policytype="policy" poluuid="XXXXX" service="tcp/8088" dstcountry="Australia" srccountry="Netherlands" trandisp="noop" duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 appcat="unscanned" crscore=30 craction=131072 crlevel="high" mastersrcmac="XXXXX" srcmac="XXXXX" srcserver=0

When i use btool it looks like the correct props are being applied

D:\Program Files\Splunk\bin>splunk btool props list | findstr FWCL01
[host::FWCL01]
TRANSFORMS-set_null = FWCL01_ruleid0_to_null, FWCL01_ruleid4_to_null

 

Any idea's?

0 Karma

PickleRick
SplunkTrust
SplunkTrust

At first glance looks pretty OK. Are you 100% sure the host value is right? (bonus question - isn't the host value extracted and overwritten in transforms?)

0 Karma

nick_currie
Path Finder

Aha - OK this might be where I am going wrong. The host is right - but I cant see the host field within the event log entry when i look at the source.. Is this why its not triggering? do I need to use devname field devname="FWCL01"?

 

These logs are sent from a Fortianalyzer to a syslog - so perhaps the Host value is generated in a different part of the process

 

 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

The question is how you're getting that data. Typically the host is either set for a specific input, or might be (for example with HEC) pushed by the source with the event data.

0 Karma

nick_currie
Path Finder

Sorry bear with me here - i have inherited this environment and am a splunk n00b -

 

So it looks like we have the Splunk_TA_fortinet_fortigate app installed and this generates the hostname from the devname based on the transforms.conf file in that app as shown below: does this mean i cannot filter on HF's based on the host value?

 

##sourcetype
[force_sourcetype_fgt]
SOURCE_KEY = _raw
DEST_KEY = MetaData:Sourcetype
REGEX = ^.+?devid=\"?F(?:G|W|6K).+?(?:\s|\,|\,\s)type=\"?(traffic|utm|event|anomaly)
FORMAT = sourcetype::fgt_$1


[fgt_change_hostname]
SOURCE_KEY = _raw
DEST_KEY = MetaData:Host
REGEX = ^.+?devname=\"(\S+)\"\s
FORMAT = host::$1


## LOOKUP

[ftnt_protocol_lookup]
filename = ftnt_protocol_info.csv

[ftnt_action_lookup]
filename = ftnt_action_info.csv

[ftnt_event_action_lookup]
filename = ftnt_event_action_info.csv

## REPORT

[field_extract]
DELIMS = "\ ,", "="

 

Tags (1)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

Apparently the priorities are so that [source::*] pattern settings are applied first, then [host::*] and at the end the general sourcetype settings. And the resulting settings to be applied are decided as far as I remember with the values at the beginning of the parsing/transforming process (so that overwritten field values are not taken into account), you can'd match to this value of yours. (as a side trivia - you cannot make a loop with overwriting metadata; I tried ;-)). So you have to either attach your transforms to the sourcetype-level settings or check for the original host field value, before rewriting. It will most probably be either set on the input or will come from the hostname of the forwarder getting the events from your fortigate devices.

0 Karma

nick_currie
Path Finder

Thank you very much for all your help Rick! Unfortunately the original host is a syslog server that has a few different input files - however the file that holds all the forti events is a single input as it's aggregated byour Fortianalyzer device.

Plan B i think will have to be a fairly lengthy regexp that has both the policy ID and deviceid. Our Heavy Forwarders have resonable processing power however they are already sitting around 50% util - hopefully this extra pattern matching will not create too much of an overhead.

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Can’t Make It to Boston? Stream .conf25 and Learn with Haya Husain

Boston may be buzzing this September with Splunk University and .conf25, but you don’t have to pack a bag to ...

Splunk Lantern’s Guide to The Most Popular .conf25 Sessions

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Unlock What’s Next: The Splunk Cloud Platform at .conf25

In just a few days, Boston will be buzzing as the Splunk team and thousands of community members come together ...