nullQueue question - Palo Logs

[danielfurtaw]

Hi Splunk folks,

My team is seeing a pesky issue with Palo Alto logs where a small subset are not being sourcetyped into pan:traffic/threat, etc. As the pan:log is the default, we have a few logs that keep this sourcetype. We have attempted to regex and nullQueue out the remainder of the pan:log logs, but no success. When we implement this TRANSFORMS/Props.conf entry, we place it at the end as we understood the order followed a left to right priority.

Example of a log that is being sourcetyped as "pan:log", and we would want to drop. It seems as if this is fragmented from Syslog, but nonetheless, junk to us.

000-1823048e98,0,0,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,2021-11-10T10:04:03.905+00:00,,,infrastructure,networking,network-protocol,3,"used-by-malware,has-known-vulnerability,pervasive-use",,dns,no,no,0

Palo Alto props.conf:

[pan_log]
pulldown_type = false
SHOULD_LINEMERGE = false
TIME_PREFIX = ^(?:[^,]*,){5}
MAX_TIMESTAMP_LOOKAHEAD = 100
TRANSFORMS-sourcetype = pan_threat, pan_traffic, pan_system, pan_config, pan_hipmatch, pan_correlation, pan_userid, pan_globalprotect, pan_decryption

We added a new nullQueue entry into the transforms.conf and then inserted the pan_discard after the last entry in the transforms above.

[pan_discard]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

Any suggestions?

[bhargavi]

Hi @danielfurtaw ,

Nullqueue entry in transforms.conf looks fine.

As far as I know, entry for TRANSFORMS- in the props.conf only matters. It executes from Right to Left.
Order does not matter in the transforms.conf file.

i.e. From the below entries, first pan_decryption is executed, then pan_globalprotect and so on. 

TRANSFORMS-sourcetype = pan_threat, pan_traffic, pan_system, pan_config, pan_hipmatch, pan_correlation, pan_userid, pan_globalprotect, pan_decryption

So if you have the regexes ready for all the below transform segments..  you need to place pan_discard in the first.

TRANSFORMS-sourcetype = pan_discard ,pan_threat, pan_traffic, pan_system, pan_config, pan_hipmatch, pan_correlation, pan_userid, pan_globalprotect, pan_decryption

So after all the regexes are executed, pan_discard comes into picture and the remaining logs will be sent to nullQueue.

Try the above and let me know . If its helps, please give an upvote 🙂

Happy Splunking 🙂