topic Re: Using props.conf and transforms.conf to exclude 'USERID' events in Palo Alto logs in Getting Data In

Using props.conf and transforms.conf to exclude 'USERID' events in Palo Alto logs

km1986 — Mon, 07 Sep 2020 14:29:22 GMT

Hello All, I'm trying to prevent the 'USERID' events from getting indexed by making the following changes on my Heavy Forwarder. However, after adding the TRANSFORMS-null statement and the [setnull] stanza in transforms.conf, I'm not seeing any logs getting indexed at all. Any guidance is appreciated

inputs.conf

[monitor:///var/log/palo]
disabled = false
sourcetype = pan:traffic

props.conf

[pan:traffic]
TRANSFORMS-null = setnull
TZ = America/New_York
TRANSFORMS-host = paloalto-host
DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = true
disabled = false
pulldown_type = true

transforms.conf

[paloalto-host]
SOURCE_KEY = _raw
FORMAT = host::$1
DEST_KEY = MetaData:Host

[setnull]
REGEX = ^(?:[^,\n]*,){3}USERID
DEST_KEY = queue
FORMAT = nullQueue

Re: Using props.conf and transforms.conf to exclude 'USERID' events in Palo Alto logs

gcusello — Mon, 07 Sep 2020 14:43:51 GMT

HBi @km1986,

wher do you located the files?

thet must stay on the Indexers or (when present) on Heavy Forwarders.

Are you sure about the regex? if you share a sample of your logs I could help you in this check..

Ciao.

Giuseppe

Re: Using props.conf and transforms.conf to exclude 'USERID' events in Palo Alto logs

km1986 — Mon, 07 Sep 2020 15:01:16 GMT

Hey @gcusello

The files are located on the Heavy Forwarder (/var/log/palo). I have double-checked the regex, it seems to be fine. None of the logs are getting indexed at all, even the ones not of 'USERID' type, which is why I was thinking if something is wrong in the props/transforms?

Below is a sample:

Sep 7 03:29:28 ttt-tt-ttt-9 1,2020/09/07 03:29:28,000000000000000,USERID,end,2304,2020/09/07 03:29:18,172.17.132.5,172.17.130.68,0.0.0.0,0.0.0.0,tttttt-tttttttt,,,dns,vsys1,trust,trust,ethernet1/2,ethernet1/2,default,2020/09/07 03:29:18,386215,1,50473,53,0,0,0x64,udp,allow,260,102,158,2,202 0/09/07 03:28:47,0,any,0,10906416,0x8000000000000000,ttt-tt-ttttt-ttttt,ttt-tt-ttttt-ttttt,0,1,1,aged-out,324,327,0,0,,tttttttttttt,from-policy,,,0,,0,,N/A,0,0,0,0,tttttttt-tttt-tttt-tttt-tttttttttttt,0

Re: Using props.conf and transforms.conf to exclude 'USERID' events in Palo Alto logs

gcusello — Mon, 07 Sep 2020 15:22:12 GMT

Hi @km1986,

it's difficoult to see you regex: please use the "Insert/Edit Code Sample" button when you have code:

Anyway, the problem is that all the logs are filtered, is it correct?

you could try using a simpler regex

REGEX = USERID

and

a more complete props and transforms:

pros.conf:

[pan:traffic] TRANSFORMS-set= setparsing,setnull

transforms.conf

[setnull] REGEX = USERID DEST_KEY = queue FORMAT = nullQueue [setparsing] REGEX = . DEST_KEY = queue FORMAT = indexQueue

Then I see that the host override is missing of a command:

[paloalto-host] REGEX = . FORMAT = host::$1 DEST_KEY = MetaData:Host

Ciao.

Giuseppe

Re: Using props.conf and transforms.conf to exclude 'USERID' events in Palo Alto logs

km1986 — Thu, 24 Sep 2020 16:41:46 GMT

Thanks @gcusello, this worked. I think I had an issue with the splunk test instance which was not indexing logs properly since I was seeing issues with some other logs as well.

I spun up a fresh instance and tried it and it worked.

Re: Using props.conf and transforms.conf to exclude 'USERID' events in Palo Alto logs

gcusello — Fri, 25 Sep 2020 06:40:51 GMT

Hi @km1986,

Good for you.

Ciao and happy splunking.

Giuseppe

P.S.: Karma Points are appreciated 😉

Re: Using props.conf and transforms.conf to exclude 'USERID' events in Palo Alto logs

sureshkumaar — Mon, 17 Feb 2025 06:06:46 GMT

Hi @gcusello - Will this work if we give some more values to be considered for indexing in transforms.conf?

[setparsing]
REGEX = systemd | auditd | CROND
DEST_KEY = queue
FORMAT = indexQueue

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

Re: Using props.conf and transforms.conf to exclude 'USERID' events in Palo Alto logs

gcusello — Mon, 17 Feb 2025 08:21:55 GMT

Hi @sureshkumaar ,

it isn't a good idea to attach a new question to a closed question, even if on the same topic: it's always better to open a new one to have a surely faster and probably better answer to your question.

Anyway, if the regex that you're using matches all the events to filter, it's correct and you can use it.

Ciao.

Giuseppe