Splunk Search

How to capture URL information

bagarwal
Path Finder

Hello All,

I want to create a report for top 10 URL's visited by the users. However, when I see the events in PaloAlto Firewall , I don't see any fields containing URL information though there is URL category field.

e.g. in URL category field I am getting as "computer -and internet-info" , but I want specific URL information e.g. *.dell.com or *.net or *.saas.hp.com/ something like this.

Can any please help how to get the URL information in firewall events so I can pull the data and create the report.

Thanks in advance

Binay Agarwal

Tags (3)
0 Karma

btorresgil
Builder

Hello,

To get URL's in Splunk from a Palo Alto Networks Next-generation Firewall, you need to send URL logs to Splunk:

  1. Install a URL-Filtering license on the firewall
  2. Create a URL-Filtering security profile with all categories set to 'alert' or some other action besides 'allow' (allow does not produce a log)
  3. Assign the URL-Filtering profile to a security rule that sees the traffic you want to log.
  4. Assign the Log Forwarding profile you created for Splunk to the same rule.
  5. Commit the configuration
  6. Assuming you installed the Palo Alto Networks Add-on for Splunk, view the URL logs with this search:

    eventtype=pan log_subtype=url | table dest_hostname url

pj
Contributor

To add -

In order to forward URL logs, it is necessary to forward Threat logs of Severity 'informational' to the Syslog server on the PaloAlto server.

0 Karma

bagarwal
Path Finder

Hi @btorresgil,

Thank You for your response. Will try this also 🙂 However, would be prefer to get the URL links and view without using Palo Alto Networks App.

Thanks & Regards,
Binay Agarwal

0 Karma

btorresgil
Builder

Hi Binay, you don't need to use the App, just the Add-on. The Add-on simply contains an optimized props.conf and transforms.conf for parsing the default Palo Alto Networks logs. It will not slow down your Splunk instance, it just does all the parsing work for you so you don't have to create a parser or a custom log format. Creating a regex yourself would by much slower to process every log than the methods used in the Add-on.

Palo Alto Networks Add-on:
https://splunkbase.splunk.com/app/2757/

0 Karma

skoelpin
SplunkTrust
SplunkTrust

Hello @bagarwal, you will need to extract the field using a regular expression. Post a sample of your data and I will help you write the search

0 Karma

bagarwal
Path Finder

Hello Skoelpin,

Thank You for your response.

Here is the 2 sample data: Just have replaced some information with <>.

Hope it helps to extract the URL field using a regular expression . If not, please let me know any specific sample you need.

========================
2016-10-25T10:57:02+00:00 Palo Alto Networks|PAN-OS Syslog Integration|4.0|
deny|cat=TRAFFIC|src=|dst=|srcPort=<>|dstPort=23|proto=tcp|usrName=|
SerialNumber=007801003272|Type=TRAFFIC|Subtype=drop|srcPostNAT=0.0.0.0|dstPostNAT=0.0.0.0|RuleName=DENY-ALL|
SourceUser=|DestinationUser=|Application=not-applicable| VirtualSystem=<>|SourceZone=internet|DestinationZone=public03|
IngressInterface=<>|EgressInterface=|LogForwardingProfile=log-all-to-panorama-and-ext|SessionID=0|RepeatCount=1|srcPostNATPort=0|
dstPostNATPort=0|Flags=0x0|totalBytes=64|totalPackets=1|ElapsedTime=0|URLCategory=any|dstBytes=0|srcBytes=64|action=deny

========================================================

2016-10-25T10:57:02+00:00 Palo Alto Networks|PAN-OS Syslog Integration|4.0|allow|cat=TRAFFIC|src=|dst=|srcPort=<>|dstPort=443|proto=tcp|usrName=| SerialNumber=007801003272|Type=TRAFFIC|Subtype=end|srcPostNAT=|dstPostNAT=|RuleName=5-1|SourceUser=|DestinationUser=|Application=google-base| VirtualSystem=vsys1|SourceZone=office|DestinationZone=internet|IngressInterface=ae2.431|EgressInterface=ae1.633|LogForwardingProfile=log-all-to-panorama-and-ext|SessionID=76241|RepeatCount=1|
srcPostNATPort=<>|dstPostNATPort=443|Flags=0x40001a|totalBytes=2067|totalPackets=18|ElapsedTime=126|URLCategory=search-engines|dstBytes=770|srcBytes=1297|action=allow

Thanks & Regards,
Binay Agarwal

0 Karma

bagarwal
Path Finder

Hello @skoelpin ,

Can you please help in writing the regex or do you need any more details.

Thanks & Regards,
Binay Agarwal

0 Karma
Get Updates on the Splunk Community!

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...

DevSecOps: Why You Should Care and How To Get Started

 WATCH NOW In this Tech Talk we will talk about what people mean by DevSecOps and deep dive into the different ...