All Apps and Add-ons

Data Latency with Palo Alto Network Logs

Ellen
Splunk Employee
Splunk Employee

I have noticed that my data is indexing correctly but seems to be getting indexed more slowly than before.
I have not upgraded and have used SoS to check the indexing performance.

It looks like the Parsing, Aggregation and Typing queues are all between 75% and 100% full.

The Indexing queue is not hitting the max and nearly empty.

Total CPU usage is showing regexreplacement is between 75% and 100%.

I have the Palo Alto Networks app installed and enabled with the bulk of my data coming from the Palo Alto Network product logs. Can the regexreplacement performance be improved?

1 Solution

bpaul_splunk
Splunk Employee
Splunk Employee

There are four regular expressions in the Palo Alto Networks
default application that can be made more efficient.  The file where these
expressions are located is:

$SPLUNK_HOME/etc/apps/SplunkforPaloAltoNetworks/default/transforms.conf

The stanzas are the following:

[pan_threat]
DEST_KEY = MetaData:Sourcetype
REGEX = ([^,]+,[^,]+,[^,]+,THREAT,)
FORMAT = sourcetype::pan_threat

[pan_traffic]
DEST_KEY = MetaData:Sourcetype
REGEX = ([^,]+,[^,]+,[^,]+,TRAFFIC,)
FORMAT = sourcetype::pan_traffic

[pan_system]
DEST_KEY = MetaData:Sourcetype
REGEX = ([^,]+,[^,]+,[^,]+,SYSTEM,)
FORMAT = sourcetype::pan_system

[pan_config]
DEST_KEY = MetaData:Sourcetype
REGEX = ([^,]+,[^,]+,[^,]+,CONFIG,)
FORMAT = sourcetype::pan_config

Due to the format of the log file, this results in slow performance.  Comment
out the original REGEX  entries, and add a new line with the following as
appropriate for your environment:

REGEX = ^[^,]+,[^,]+,[^,]+,THREAT,
REGEX = ^[^,]+,[^,]+,[^,]+,TRAFFIC,
REGEX = ^[^,]+,[^,]+,[^,]+,SYSTEM,
REGEX = ^[^,]+,[^,]+,[^,]+,CONFIG,

This should optimize performance and reduce the load on the queues. 
The author of the Palo Alto Networks app is aware of this, and we hope to see
these optimized expressions in a future release.  For the latest Palo Alto
Network App, please click here.

View solution in original post

ckurtz
Path Finder

You should copy the file to local and modify it there so an incremental release before the author fixes it doesn't wipe out your changes.

I also suggest breaking your logfiles into as small of chunks as you can. We had to have our rsyslog server rotate every 15m to handle our ~400gb/day of logs, but that helped immensely.

bpaul_splunk
Splunk Employee
Splunk Employee

There are four regular expressions in the Palo Alto Networks
default application that can be made more efficient.  The file where these
expressions are located is:

$SPLUNK_HOME/etc/apps/SplunkforPaloAltoNetworks/default/transforms.conf

The stanzas are the following:

[pan_threat]
DEST_KEY = MetaData:Sourcetype
REGEX = ([^,]+,[^,]+,[^,]+,THREAT,)
FORMAT = sourcetype::pan_threat

[pan_traffic]
DEST_KEY = MetaData:Sourcetype
REGEX = ([^,]+,[^,]+,[^,]+,TRAFFIC,)
FORMAT = sourcetype::pan_traffic

[pan_system]
DEST_KEY = MetaData:Sourcetype
REGEX = ([^,]+,[^,]+,[^,]+,SYSTEM,)
FORMAT = sourcetype::pan_system

[pan_config]
DEST_KEY = MetaData:Sourcetype
REGEX = ([^,]+,[^,]+,[^,]+,CONFIG,)
FORMAT = sourcetype::pan_config

Due to the format of the log file, this results in slow performance.  Comment
out the original REGEX  entries, and add a new line with the following as
appropriate for your environment:

REGEX = ^[^,]+,[^,]+,[^,]+,THREAT,
REGEX = ^[^,]+,[^,]+,[^,]+,TRAFFIC,
REGEX = ^[^,]+,[^,]+,[^,]+,SYSTEM,
REGEX = ^[^,]+,[^,]+,[^,]+,CONFIG,

This should optimize performance and reduce the load on the queues. 
The author of the Palo Alto Networks app is aware of this, and we hope to see
these optimized expressions in a future release.  For the latest Palo Alto
Network App, please click here.

Get Updates on the Splunk Community!

How I instrumented a Rust application without knowing Rust

As a technical writer, I often have to edit or create code snippets for Splunk's distributions of ...

Splunk Community Platform Survey

Hey Splunk Community, Starting today, the community platform may prompt you to participate in a survey. The ...

Observability Highlights | November 2022 Newsletter

 November 2022Observability CloudEnd Of Support Extension for SignalFx Smart AgentSplunk is extending the End ...