Custom Log Format | Parsing issues

[brenthelm]

We are wanting to cut down on the amount of data that is going to Splunk from our Palo Alto Firewalls. In order to do that, we want to trim the unnecessary data from the logs but still have it parse correctly in Splunk. When we create the custom log format it will no longer be recognized as PAN:Traffic, instead it is being parsed as PAN:Firewall. We used the custom format from Palo Altos website and included the commas where they were supposed to go. BTW this is configured on Panorama in syslog settings. 

Before: From Palo Alto WebSite

FUTURE_USE, Receive Time, Serial Number, Type, Threat/Content Type, FUTURE_USE, Generated Time, Source Address, Destination Address, NAT Source IP, NAT Destination IP, Rule Name, Source User, Destination User, Application, Virtual System, Source Zone, Destination Zone, Inbound Interface, Outbound Interface, Log Action, FUTURE_USE, Session ID, Repeat Count, Source Port, Destination Port, NAT Source Port, NAT Destination Port, Flags, Protocol, Action, Bytes, Bytes Sent, Bytes Received, Packets, Start Time, Elapsed Time, Category, FUTURE_USE, Sequence Number, Action Flags, Source Location, Destination Location, FUTURE_USE, Packets Sent, Packets Received, Session End Reason, Device Group Hierarchy Level 1, Device Group Hierarchy Level 2, Device Group Hierarchy Level 3, Device Group Hierarchy Level 4, Virtual System Name, Device Name, Action Source, Source VM UUID, Destination VM UUID, Tunnel ID/IMSI, Monitor Tag/IMEI, Parent Session ID, Parent Start Time, Tunnel Type, SCTP Association ID, SCTP Chunks, SCTP Chunks Sent, SCTP Chunks Received, Rule UUID, HTTP/2 Connection, App Flap Count, Policy ID, Link Switches, SD-WAN Cluster, SD-WAN Device Type, SD-WAN Cluster Type, SD-WAN Site, Dynamic User Group Name

What we want:

,$receive_time,,$type,$subtype,,$time_generated,$src,$dst,$natsrc,$natdst,$rule,$srcuser,$dstuser,$app,,$to,$from,$inbound_if,$outbound_if,,,,$repeatcnt,$sport,$dport,$natsport,$natdport,$flags,$proto,$action,$bytes,$bytes_sent,$bytes_received,$packets,,,$category,,$seqno,,$srcloc,$dstloc,,$pkts_sent,$pkts_received,$session_end_reason,,,,,,$device_name,$action_source,,,,,,,,,,,,,,,,,,,,,,

We have even captured packets and compared what we are getting with what is expected and they seem to match up. Not sure what is wrong, but would love some help. Not sure Palo Alto will help, though we did submit a ticket to them. Splunk closed my ticket because the APP is "Vendor Supported". Any advice on doing this or any other suggestions to how anyone else is doing Palo Alto logs?

Thanks!!!

[brettw]

Hi there!  I assume you are using the Palo Alto TA.  It has a few layers to it where it recognizes patterns in the logs to classify it beyond the default pan:firewall sourcetype.

First Change

Look at the default transforms.conf.  You're going to need to change the REGEX so it matches your changed format.  Remember to put this stanza in the local folder of the TA.

[pan_traffic]
DEST_KEY = MetaData:Sourcetype
REGEX = ^[^,]+,[^,]+,[^,]+,TRAFFIC,
FORMAT = sourcetype::pan:traffic

Becomes...

[pan_traffic]
DEST_KEY = MetaData:Sourcetype
REGEX = ^[^,]+,TRAFFIC,
FORMAT = sourcetype::pan:traffic

Second Change

Next, also in transforms.conf, you'll also need to tweak this stanza to match your new format:

[extract_traffic]
DELIMS = ","
FIELDS = "future_use1","receive_time","serial_number","type","log_subtype","version","generated_time","src_ip","dest_ip","src_translated_ip","dest_translated_ip","rule","src_user","dest_user","app","vsys","src_zone","dest_zone","src_interface","dest_interface","log_forwarding_profile","future_use3","session_id","repeat_count","src_port","dest_port","src_translated_port","dest_translated_port","session_flags","transport","action","bytes","bytes_out","bytes_in","packets","start_time","duration","http_category","future_use4","sequence_number","action_flags","src_location","dest_location","future_use5","packets_out","packets_in","session_end_reason","devicegroup_level1","devicegroup_level2","devicegroup_level3","devicegroup_level4","vsys_name","dvc_name","action_source","src_vm","dest_vm","tunnel_id","tunnel_monitor_tag","tunnel_session_id","tunnel_start_time","tunnel_type"

Becomes...

[extract_traffic]
DELIMS = ","
FIELDS = "$receive_time","$type","$subtype","$time_generated","$src","$dst","$natsrc","$natdst","$rule","$srcuser","$dstuser","$app","$to","$from","$inbound_if","$outbound_if","$repeatcnt","$sport","$dport","$natsport","$natdport","$flags","$proto","$action","$bytes","$bytes_sent","$bytes_received","$packets","$category","$seqno","$srcloc","$dstloc","$pkts_sent","$pkts_received","$session_end_reason","$device_name","$action_source"