How to capture URL information

[bagarwal]

Hello All,

I want to create a report for top 10 URL's visited by the users. However, when I see the events in PaloAlto Firewall , I don't see any fields containing URL information though there is URL category field.

e.g. in URL category field I am getting as "computer -and internet-info" , but I want specific URL information e.g. *.dell.com or *.net or *.saas.hp.com/ something like this.

Can any please help how to get the URL information in firewall events so I can pull the data and create the report.

Thanks in advance

Binay Agarwal

[btorresgil]

Hello,

To get URL's in Splunk from a Palo Alto Networks Next-generation Firewall, you need to send URL logs to Splunk:

1. Install a URL-Filtering license on the firewall
2. Create a URL-Filtering security profile with all categories set to 'alert' or some other action besides 'allow' (allow does not produce a log)
3. Assign the URL-Filtering profile to a security rule that sees the traffic you want to log.
4. Assign the Log Forwarding profile you created for Splunk to the same rule.
5. Commit the configuration

6. Assuming you installed the Palo Alto Networks Add-on for Splunk, view the URL logs with this search:

   eventtype=pan log_subtype=url | table dest_hostname url

[pj]

To add -

In order to forward URL logs, it is necessary to forward Threat logs of Severity 'informational' to the Syslog server on the PaloAlto server.

[bagarwal]

Hi @btorresgil,

Thank You for your response. Will try this also 🙂 However, would be prefer to get the URL links and view without using Palo Alto Networks App.

Thanks & Regards,
Binay Agarwal

[btorresgil]

Hi Binay, you don't need to use the App, just the Add-on. The Add-on simply contains an optimized props.conf and transforms.conf for parsing the default Palo Alto Networks logs. It will not slow down your Splunk instance, it just does all the parsing work for you so you don't have to create a parser or a custom log format. Creating a regex yourself would by much slower to process every log than the methods used in the Add-on.

Palo Alto Networks Add-on:
https://splunkbase.splunk.com/app/2757/

[skoelpin]

Hello @bagarwal, you will need to extract the field using a regular expression. Post a sample of your data and I will help you write the search

[bagarwal]

Hello Skoelpin,

Thank You for your response.

Here is the 2 sample data: Just have replaced some information with <>.

Hope it helps to extract the URL field using a regular expression . If not, please let me know any specific sample you need.

========================
2016-10-25T10:57:02+00:00 Palo Alto Networks|PAN-OS Syslog Integration|4.0|
deny|cat=TRAFFIC|src=|dst=|srcPort=<>|dstPort=23|proto=tcp|usrName=|
SerialNumber=007801003272|Type=TRAFFIC|Subtype=drop|srcPostNAT=0.0.0.0|dstPostNAT=0.0.0.0|RuleName=DENY-ALL|
SourceUser=|DestinationUser=|Application=not-applicable| VirtualSystem=<>|SourceZone=internet|DestinationZone=public03|
IngressInterface=<>|EgressInterface=|LogForwardingProfile=log-all-to-panorama-and-ext|SessionID=0|RepeatCount=1|srcPostNATPort=0|
dstPostNATPort=0|Flags=0x0|totalBytes=64|totalPackets=1|ElapsedTime=0|URLCategory=any|dstBytes=0|srcBytes=64|action=deny

========================================================

2016-10-25T10:57:02+00:00 Palo Alto Networks|PAN-OS Syslog Integration|4.0|allow|cat=TRAFFIC|src=|dst=|srcPort=<>|dstPort=443|proto=tcp|usrName=| SerialNumber=007801003272|Type=TRAFFIC|Subtype=end|srcPostNAT=|dstPostNAT=|RuleName=5-1|SourceUser=|DestinationUser=|Application=google-base| VirtualSystem=vsys1|SourceZone=office|DestinationZone=internet|IngressInterface=ae2.431|EgressInterface=ae1.633|LogForwardingProfile=log-all-to-panorama-and-ext|SessionID=76241|RepeatCount=1|
srcPostNATPort=<>|dstPostNATPort=443|Flags=0x40001a|totalBytes=2067|totalPackets=18|ElapsedTime=126|URLCategory=search-engines|dstBytes=770|srcBytes=1297|action=allow

Thanks & Regards,
Binay Agarwal

[bagarwal]

Hello @skoelpin ,

Can you please help in writing the regex or do you need any more details.

Thanks & Regards,
Binay Agarwal