All Apps and Add-ons

Data Latency with Palo Alto Network Logs

Ellen
Splunk Employee
Splunk Employee

I have noticed that my data is indexing correctly but seems to be getting indexed more slowly than before.
I have not upgraded and have used SoS to check the indexing performance.

It looks like the Parsing, Aggregation and Typing queues are all between 75% and 100% full.

The Indexing queue is not hitting the max and nearly empty.

Total CPU usage is showing regexreplacement is between 75% and 100%.

I have the Palo Alto Networks app installed and enabled with the bulk of my data coming from the Palo Alto Network product logs. Can the regexreplacement performance be improved?

1 Solution

bpaul_splunk
Splunk Employee
Splunk Employee

There are four regular expressions in the Palo Alto Networks
default application that can be made more efficient.  The file where these
expressions are located is:

$SPLUNK_HOME/etc/apps/SplunkforPaloAltoNetworks/default/transforms.conf

The stanzas are the following:

[pan_threat]
DEST_KEY = MetaData:Sourcetype
REGEX = ([^,]+,[^,]+,[^,]+,THREAT,)
FORMAT = sourcetype::pan_threat

[pan_traffic]
DEST_KEY = MetaData:Sourcetype
REGEX = ([^,]+,[^,]+,[^,]+,TRAFFIC,)
FORMAT = sourcetype::pan_traffic

[pan_system]
DEST_KEY = MetaData:Sourcetype
REGEX = ([^,]+,[^,]+,[^,]+,SYSTEM,)
FORMAT = sourcetype::pan_system

[pan_config]
DEST_KEY = MetaData:Sourcetype
REGEX = ([^,]+,[^,]+,[^,]+,CONFIG,)
FORMAT = sourcetype::pan_config

Due to the format of the log file, this results in slow performance.  Comment
out the original REGEX  entries, and add a new line with the following as
appropriate for your environment:

REGEX = ^[^,]+,[^,]+,[^,]+,THREAT,
REGEX = ^[^,]+,[^,]+,[^,]+,TRAFFIC,
REGEX = ^[^,]+,[^,]+,[^,]+,SYSTEM,
REGEX = ^[^,]+,[^,]+,[^,]+,CONFIG,

This should optimize performance and reduce the load on the queues. 
The author of the Palo Alto Networks app is aware of this, and we hope to see
these optimized expressions in a future release.  For the latest Palo Alto
Network App, please click here.

View solution in original post

ckurtz
Path Finder

You should copy the file to local and modify it there so an incremental release before the author fixes it doesn't wipe out your changes.

I also suggest breaking your logfiles into as small of chunks as you can. We had to have our rsyslog server rotate every 15m to handle our ~400gb/day of logs, but that helped immensely.

bpaul_splunk
Splunk Employee
Splunk Employee

There are four regular expressions in the Palo Alto Networks
default application that can be made more efficient.  The file where these
expressions are located is:

$SPLUNK_HOME/etc/apps/SplunkforPaloAltoNetworks/default/transforms.conf

The stanzas are the following:

[pan_threat]
DEST_KEY = MetaData:Sourcetype
REGEX = ([^,]+,[^,]+,[^,]+,THREAT,)
FORMAT = sourcetype::pan_threat

[pan_traffic]
DEST_KEY = MetaData:Sourcetype
REGEX = ([^,]+,[^,]+,[^,]+,TRAFFIC,)
FORMAT = sourcetype::pan_traffic

[pan_system]
DEST_KEY = MetaData:Sourcetype
REGEX = ([^,]+,[^,]+,[^,]+,SYSTEM,)
FORMAT = sourcetype::pan_system

[pan_config]
DEST_KEY = MetaData:Sourcetype
REGEX = ([^,]+,[^,]+,[^,]+,CONFIG,)
FORMAT = sourcetype::pan_config

Due to the format of the log file, this results in slow performance.  Comment
out the original REGEX  entries, and add a new line with the following as
appropriate for your environment:

REGEX = ^[^,]+,[^,]+,[^,]+,THREAT,
REGEX = ^[^,]+,[^,]+,[^,]+,TRAFFIC,
REGEX = ^[^,]+,[^,]+,[^,]+,SYSTEM,
REGEX = ^[^,]+,[^,]+,[^,]+,CONFIG,

This should optimize performance and reduce the load on the queues. 
The author of the Palo Alto Networks app is aware of this, and we hope to see
these optimized expressions in a future release.  For the latest Palo Alto
Network App, please click here.

Get Updates on the Splunk Community!

Splunk at Cisco Live 2025: Learning, Innovation, and a Little Bit of Mr. Brightside

Pack your bags (and maybe your dancing shoes)—Cisco Live is heading to San Diego, June 8–12, 2025, and Splunk ...

Splunk App Dev Community Updates – What’s New and What’s Next

Welcome to your go-to roundup of everything happening in the Splunk App Dev Community! Whether you're building ...

The Latest Cisco Integrations With Splunk Platform!

Join us for an exciting tech talk where we’ll explore the latest integrations in Cisco + Splunk! We’ve ...