Data Latency with Palo Alto Network Logs

[Ellen]

I have noticed that my data is indexing correctly but seems to be getting indexed more slowly than before.
I have not upgraded and have used SoS to check the indexing performance.

It looks like the Parsing, Aggregation and Typing queues are all between 75% and 100% full.

The Indexing queue is not hitting the max and nearly empty.

Total CPU usage is showing regexreplacement is between 75% and 100%.

I have the Palo Alto Networks app installed and enabled with the bulk of my data coming from the Palo Alto Network product logs. Can the regexreplacement performance be improved?

[ckurtz]

You should copy the file to local and modify it there so an incremental release before the author fixes it doesn't wipe out your changes.

I also suggest breaking your logfiles into as small of chunks as you can. We had to have our rsyslog server rotate every 15m to handle our ~400gb/day of logs, but that helped immensely.

[bpaul_splunk · Accepted solution]

There are four regular expressions in the Palo Alto Networks
default application that can be made more efficient.  The file where these
expressions are located is:

$SPLUNK_HOME/etc/apps/SplunkforPaloAltoNetworks/default/transforms.conf

The stanzas are the following:

[pan_threat]
DEST_KEY = MetaData:Sourcetype
REGEX = ([^,]+,[^,]+,[^,]+,THREAT,)
FORMAT = sourcetype::pan_threat

[pan_traffic]
DEST_KEY = MetaData:Sourcetype
REGEX = ([^,]+,[^,]+,[^,]+,TRAFFIC,)
FORMAT = sourcetype::pan_traffic

[pan_system]
DEST_KEY = MetaData:Sourcetype
REGEX = ([^,]+,[^,]+,[^,]+,SYSTEM,)
FORMAT = sourcetype::pan_system

[pan_config]
DEST_KEY = MetaData:Sourcetype
REGEX = ([^,]+,[^,]+,[^,]+,CONFIG,)
FORMAT = sourcetype::pan_config

Due to the format of the log file, this results in slow performance.  Comment
out the original REGEX  entries, and add a new line with the following as
appropriate for your environment:

REGEX = ^[^,]+,[^,]+,[^,]+,THREAT,
REGEX = ^[^,]+,[^,]+,[^,]+,TRAFFIC,
REGEX = ^[^,]+,[^,]+,[^,]+,SYSTEM,
REGEX = ^[^,]+,[^,]+,[^,]+,CONFIG,

This should optimize performance and reduce the load on the queues. 
The author of the Palo Alto Networks app is aware of this, and we hope to see
these optimized expressions in a future release.  For the latest Palo Alto
Network App, please click here.