All Apps and Add-ons

Data Latency with Palo Alto Network Logs

Ellen
Splunk Employee
Splunk Employee

I have noticed that my data is indexing correctly but seems to be getting indexed more slowly than before.
I have not upgraded and have used SoS to check the indexing performance.

It looks like the Parsing, Aggregation and Typing queues are all between 75% and 100% full.

The Indexing queue is not hitting the max and nearly empty.

Total CPU usage is showing regexreplacement is between 75% and 100%.

I have the Palo Alto Networks app installed and enabled with the bulk of my data coming from the Palo Alto Network product logs. Can the regexreplacement performance be improved?

1 Solution

bpaul_splunk
Splunk Employee
Splunk Employee

There are four regular expressions in the Palo Alto Networks
default application that can be made more efficient.  The file where these
expressions are located is:

$SPLUNK_HOME/etc/apps/SplunkforPaloAltoNetworks/default/transforms.conf

The stanzas are the following:

[pan_threat]
DEST_KEY = MetaData:Sourcetype
REGEX = ([^,]+,[^,]+,[^,]+,THREAT,)
FORMAT = sourcetype::pan_threat

[pan_traffic]
DEST_KEY = MetaData:Sourcetype
REGEX = ([^,]+,[^,]+,[^,]+,TRAFFIC,)
FORMAT = sourcetype::pan_traffic

[pan_system]
DEST_KEY = MetaData:Sourcetype
REGEX = ([^,]+,[^,]+,[^,]+,SYSTEM,)
FORMAT = sourcetype::pan_system

[pan_config]
DEST_KEY = MetaData:Sourcetype
REGEX = ([^,]+,[^,]+,[^,]+,CONFIG,)
FORMAT = sourcetype::pan_config

Due to the format of the log file, this results in slow performance.  Comment
out the original REGEX  entries, and add a new line with the following as
appropriate for your environment:

REGEX = ^[^,]+,[^,]+,[^,]+,THREAT,
REGEX = ^[^,]+,[^,]+,[^,]+,TRAFFIC,
REGEX = ^[^,]+,[^,]+,[^,]+,SYSTEM,
REGEX = ^[^,]+,[^,]+,[^,]+,CONFIG,

This should optimize performance and reduce the load on the queues. 
The author of the Palo Alto Networks app is aware of this, and we hope to see
these optimized expressions in a future release.  For the latest Palo Alto
Network App, please click here.

View solution in original post

ckurtz
Path Finder

You should copy the file to local and modify it there so an incremental release before the author fixes it doesn't wipe out your changes.

I also suggest breaking your logfiles into as small of chunks as you can. We had to have our rsyslog server rotate every 15m to handle our ~400gb/day of logs, but that helped immensely.

bpaul_splunk
Splunk Employee
Splunk Employee

There are four regular expressions in the Palo Alto Networks
default application that can be made more efficient.  The file where these
expressions are located is:

$SPLUNK_HOME/etc/apps/SplunkforPaloAltoNetworks/default/transforms.conf

The stanzas are the following:

[pan_threat]
DEST_KEY = MetaData:Sourcetype
REGEX = ([^,]+,[^,]+,[^,]+,THREAT,)
FORMAT = sourcetype::pan_threat

[pan_traffic]
DEST_KEY = MetaData:Sourcetype
REGEX = ([^,]+,[^,]+,[^,]+,TRAFFIC,)
FORMAT = sourcetype::pan_traffic

[pan_system]
DEST_KEY = MetaData:Sourcetype
REGEX = ([^,]+,[^,]+,[^,]+,SYSTEM,)
FORMAT = sourcetype::pan_system

[pan_config]
DEST_KEY = MetaData:Sourcetype
REGEX = ([^,]+,[^,]+,[^,]+,CONFIG,)
FORMAT = sourcetype::pan_config

Due to the format of the log file, this results in slow performance.  Comment
out the original REGEX  entries, and add a new line with the following as
appropriate for your environment:

REGEX = ^[^,]+,[^,]+,[^,]+,THREAT,
REGEX = ^[^,]+,[^,]+,[^,]+,TRAFFIC,
REGEX = ^[^,]+,[^,]+,[^,]+,SYSTEM,
REGEX = ^[^,]+,[^,]+,[^,]+,CONFIG,

This should optimize performance and reduce the load on the queues. 
The author of the Palo Alto Networks app is aware of this, and we hope to see
these optimized expressions in a future release.  For the latest Palo Alto
Network App, please click here.

Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Take Action Automatically on Splunk Alerts with Red Hat Ansible Automation Platform

 Are you ready to revolutionize your IT operations? As digital transformation accelerates, the demand for ...

Calling All Security Pros: Ready to Race Through Boston?

Hey Splunkers, .conf25 is heading to Boston and we’re kicking things off with something bold, competitive, and ...

Beyond Detection: How Splunk and Cisco Integrated Security Platforms Transform ...

Financial services organizations face an impossible equation: maintain 99.9% uptime for mission-critical ...