Filtration of logs from Particular sourcetypes

[anandhalagaras1]

Hi Team,

We are ingesting Palo Alto Firewall logs into Splunk from our Syslog server. Hence we have made the Syslog server as a Heavy Forwarder as well.

And in the Syslog Heavy Forwarder server we have installed the "Splunk_TA_paloalto" Add-on and configured the inputs as "pan:firewall" so that based on the TA the data is getting segregated with different sourcetypes such as "pan:hipmatch" "pan:userid" , "pan:system" ,"pan:traffic" & "pan:threat".

So now we want to filter out the log ingestion from few of the sourcetypes such as "pan:hipmatch", "pan:userid" & "pan:system" alone. So how to filter those logs from Splunk before ingestion.

Where should i need to place the props and transforms and what would be the props and transforms to filter those logs from those particular sourcetypes?

Hence kindly help on my request.

[scelikok]

Hi @anandhalagaras1,

If you are ingesting logs to two different indexes, you should be using to inputs. You can use the same config that I proposed only by changing the props stanza with source.

As a sample; assuming you are getting pan log via syslog server using file monitor /log/syslog/pan; 

props.conf
[source::/log/syslog/pan]
TRANSFORMS-filter = setnull

transforms.conf
[setnull]
REGEX = ^[^,]+,[^,]+,[^,]+,HIPMATCH|SYSTEM|USERID,
DEST_KEY = queue
FORMAT = nullQueue

[scelikok · Accepted solution]

Hi @anandhalagaras1,

You should place below props snd transforms configs to the heavy forwarder.  You can play with the regex to filter other events too.

props.conf
[pan:firewall]
TRANSFORMS-filter = setnull

transforms.conf
[setnull]
REGEX = ^[^,]+,[^,]+,[^,]+,HIPMATCH|SYSTEM|USERID,
DEST_KEY = queue
FORMAT = nullQueue

[anandhalagaras1]

@scelikok ,

Thank you for your help. And now i have resolved the issue from my end post which the logs are not getting ingested into Splunk for these sourcetypes.

[anandhalagaras1]

@scelikok ,

As you mentioned I have used to filter out the logs based on sourcetype "pan:firewall" and placed it in our heavy forwarder server but still i can see the logs are getting ingested into splunk from pan:userid, pan:hipmatch & pan:system. Is it anything related to case sensitive?

As per the Add-On "Splunk_TA_paloalto" it is mentioned as below in the props.conf:

[pan:firewall]
category = Network & Security
description = Syslog from Palo Alto Networks Next-generation Firewall
pulldown_type = true
SHOULD_LINEMERGE = false
TIME_PREFIX = ^(?:[^,]*,){6}
MAX_TIMESTAMP_LOOKAHEAD = 32
TRANSFORMS-sourcetype = pan_threat, pan_traffic, pan_system, pan_config, pan_hipmatch, pan_correlation, pan_userid, pan_traps4

So kindly help me out to filter those logs from Splunk.

props.conf
[pan:firewall]
TRANSFORMS-filter = setnull

transforms.conf
[setnull]
REGEX = ^[^,]+,[^,]+,[^,]+,HIPMATCH|SYSTEM|USERID,
DEST_KEY = queue
FORMAT = nullQueue

[anandhalagaras1]

@scelikok ,

So when checked from my end,I can see that the source has been already configured with a props.conf i.e. the Timezone (TZ). So hence I have removed the props which i created with the sourcetype and added the stanza as provided in the source props which already exists and created the transforms.conf as mentioned.

props.conf:
[source::/var/log/remote/0000-Today/logfile/hostname]
TZ = CST6CDT
TRANSFORMS-filter = setnull220

transforms.conf:
[setnull220]
REGEX = ^[^,]+,[^,]+,[^,]+,HIPMATCH|SYSTEM|USERID,
DEST_KEY = queue
FORMAT = nullQueue

But still i can see the logs are getting ingested into Splunk with the following sourcetype pan:userid, pan:system and pan:hipmatch.

The extracted field names for HIPMATCH, SYSTEM and USERID comes under "type".

So kindly let me know where is the exact error & how can i stop ingesting those logs before ingestion into Splunk.

[anandhalagaras1]

@scelikok ,

Also I have tried the regex like mentioned below as well but still the logs are getting ingested into Splunk.

props.conf:
[source::/var/log/remote/0000-Today/logfile/hostname]
TZ = CST6CDT
TRANSFORMS-filter = setnull220

transforms.conf:
[setnull220]
REGEX = (?=USERID|SYSTEM|HIPMATCH) OR REGEX = (USERID|SYSTEM|HIPMATCH)
DEST_KEY = queue
FORMAT = nullQueue

But still the data is not getting filtered out before ingestion. So kindly help me on the same.

[anandhalagaras1]

Hi All,

I tried the props and transforms with sourcetype as "pan:firewall" in the Syslog HF server but it didnt worked. So now I have tried directly with the sourcetype as "pan:userid" with the following stanza but still the logs are getting ingested into Splunk.

I want to get it filtered out before ingestion itself so that we can save few licenses but unfortunately its not working.

This is my latest props and transforms which i used in the Syslog HF server for filtration.

props.conf:

[pan:userid]
TRANSFORMS-filter= setnull_case07

transforms.conf
[setnull_case07]
REGEX = ^[^,]+,[^,]+,[^,]+,USERID,
DEST_KEY = queue
FORMAT = nullQueue

So anyone kindly help on my request.

[anandhalagaras1]

@scelikok 

Thank you for your prompt response. But we have configured the pan:firewall sourcetype for two indexes.

Index= abc & def

So i want to filter out the logs only from index=abc; sourcetype=pan:firewall  so after segregation of sourcetype in that i want to filter the HIPMATCH, SYSTEM and USERID.

So kindly help on the same.