Using props.conf and transforms.conf to exclude 'USERID' events in Palo Alto logs

[km1986]

Hello All, I'm trying to prevent the 'USERID' events from getting indexed by making the following changes on my Heavy Forwarder. However, after adding the TRANSFORMS-null statement and the [setnull] stanza in transforms.conf, I'm not seeing any logs getting indexed at all. Any guidance is appreciated

inputs.conf

[monitor:///var/log/palo]
disabled = false
sourcetype = pan:traffic

props.conf

[pan:traffic]
TRANSFORMS-null = setnull
TZ = America/New_York
TRANSFORMS-host = paloalto-host
DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = true
disabled = false
pulldown_type = true

transforms.conf

[paloalto-host]
SOURCE_KEY = _raw
FORMAT = host::$1
DEST_KEY = MetaData:Host

[setnull]
REGEX = ^(?:[^,\n]*,){3}USERID
DEST_KEY = queue
FORMAT = nullQueue

[km1986]

Hey @gcusello 

The files are located on the Heavy Forwarder (/var/log/palo). I have double-checked the regex, it seems to be fine. None of the logs are getting indexed at all, even the ones not of 'USERID' type, which is why I was thinking if something is wrong in the props/transforms?

Below is a sample:

Sep 7 03:29:28 ttt-tt-ttt-9 1,2020/09/07 03:29:28,000000000000000,USERID,end,2304,2020/09/07 03:29:18,172.17.132.5,172.17.130.68,0.0.0.0,0.0.0.0,tttttt-tttttttt,,,dns,vsys1,trust,trust,ethernet1/2,ethernet1/2,default,2020/09/07 03:29:18,386215,1,50473,53,0,0,0x64,udp,allow,260,102,158,2,202 0/09/07 03:28:47,0,any,0,10906416,0x8000000000000000,ttt-tt-ttttt-ttttt,ttt-tt-ttttt-ttttt,0,1,1,aged-out,324,327,0,0,,tttttttttttt,from-policy,,,0,,0,,N/A,0,0,0,0,tttttttt-tttt-tttt-tttt-tttttttttttt,0

[gcusello · Accepted solution]

Hi @km1986,

it's difficoult to see you regex: please use the "Insert/Edit Code Sample" button when you have code:

Anyway, the problem is that all the logs are filtered, is it correct?

you could try using a simpler regex

REGEX = USERID

and

a more complete props and transforms:

pros.conf:

[pan:traffic]
TRANSFORMS-set= setparsing,setnull

transforms.conf

[setnull]
REGEX = USERID
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = .
DEST_KEY = queue
FORMAT = indexQueue

Then I see that the host override is missing of a command:

[paloalto-host]
REGEX = .
FORMAT = host::$1
DEST_KEY = MetaData:Host

Ciao.

Giuseppe

[km1986]

Thanks @gcusello, this worked. I think I had an issue with the splunk test instance which was not indexing logs properly since I was seeing issues with some other logs as well.

I spun up a fresh instance and tried it and it worked.

[gcusello]

Hi @km1986,

Good for you.

Ciao and happy splunking.

Giuseppe

P.S.: Karma Points are appreciated 😉

[gcusello]

HBi @km1986,

wher do you located the files?

thet must stay on the Indexers or (when present) on Heavy Forwarders.

Are you sure about the regex? if you share a sample of your logs  I could help you in this check.. 

Ciao.

Giuseppe