Using props.conf and transforms.conf to exclude 'USERID' events in Palo Alto logs

[km1986]

Hello All, I'm trying to prevent the 'USERID' events from getting indexed by making the following changes on my Heavy Forwarder. However, after adding the TRANSFORMS-null statement and the [setnull] stanza in transforms.conf, I'm not seeing any logs getting indexed at all. Any guidance is appreciated

inputs.conf

[monitor:///var/log/palo]
disabled = false
sourcetype = pan:traffic

props.conf

[pan:traffic]
TRANSFORMS-null = setnull
TZ = America/New_York
TRANSFORMS-host = paloalto-host
DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = true
disabled = false
pulldown_type = true

transforms.conf

[paloalto-host]
SOURCE_KEY = _raw
FORMAT = host::$1
DEST_KEY = MetaData:Host

[setnull]
REGEX = ^(?:[^,\n]*,){3}USERID
DEST_KEY = queue
FORMAT = nullQueue

[km1986]

Hey @gcusello 

The files are located on the Heavy Forwarder (/var/log/palo). I have double-checked the regex, it seems to be fine. None of the logs are getting indexed at all, even the ones not of 'USERID' type, which is why I was thinking if something is wrong in the props/transforms?

Below is a sample:

Sep 7 03:29:28 ttt-tt-ttt-9 1,2020/09/07 03:29:28,000000000000000,USERID,end,2304,2020/09/07 03:29:18,172.17.132.5,172.17.130.68,0.0.0.0,0.0.0.0,tttttt-tttttttt,,,dns,vsys1,trust,trust,ethernet1/2,ethernet1/2,default,2020/09/07 03:29:18,386215,1,50473,53,0,0,0x64,udp,allow,260,102,158,2,202 0/09/07 03:28:47,0,any,0,10906416,0x8000000000000000,ttt-tt-ttttt-ttttt,ttt-tt-ttttt-ttttt,0,1,1,aged-out,324,327,0,0,,tttttttttttt,from-policy,,,0,,0,,N/A,0,0,0,0,tttttttt-tttt-tttt-tttt-tttttttttttt,0

[gcusello · Accepted solution]

Hi @km1986,

it's difficoult to see you regex: please use the "Insert/Edit Code Sample" button when you have code:

Anyway, the problem is that all the logs are filtered, is it correct?

you could try using a simpler regex

REGEX = USERID

and

a more complete props and transforms:

pros.conf:

[pan:traffic]
TRANSFORMS-set= setparsing,setnull

transforms.conf

[setnull]
REGEX = USERID
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = .
DEST_KEY = queue
FORMAT = indexQueue

Then I see that the host override is missing of a command:

[paloalto-host]
REGEX = .
FORMAT = host::$1
DEST_KEY = MetaData:Host

Ciao.

Giuseppe

[sureshkumaar]

Hi @gcusello - Will this work if we give some more values to be considered for indexing in transforms.conf?

[setparsing]
REGEX = systemd | auditd | CROND
DEST_KEY = queue
FORMAT = indexQueue

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[gcusello]

Hi @sureshkumaar ,

it isn't a good idea to attach a new question to a closed question, even if on the same topic: it's always better to open a new one to have a surely faster and probably better answer to your question.

Anyway, if the regex that you're using matches all the events to filter, it's correct and you can use it.

Ciao.

Giuseppe

[km1986]

Thanks @gcusello, this worked. I think I had an issue with the splunk test instance which was not indexing logs properly since I was seeing issues with some other logs as well.

I spun up a fresh instance and tried it and it worked.

[gcusello]

Hi @km1986,

Good for you.

Ciao and happy splunking.

Giuseppe

P.S.: Karma Points are appreciated 😉

[gcusello]

HBi @km1986,

wher do you located the files?

thet must stay on the Indexers or (when present) on Heavy Forwarders.

Are you sure about the regex? if you share a sample of your logs  I could help you in this check.. 

Ciao.

Giuseppe