topic How to remove logs from specific sourcetype being indexed? in Getting Data In

How to remove logs from specific sourcetype being indexed?

newsplunker1 — Mon, 13 Feb 2023 23:28:10 GMT

Hi Everyone,

Im trying to stop the following index from being indexed into Splunk using the props/transforms confs on HF but with no luck - What am i doing wrong here ?

props.conf

[pan:userid]
TRANSFORMS-set-nullqueue=set_nullqueue

transforms.conf

[set_nullqueue]
REGEX=.
DEST_KEY=queue
FORMAT=nullQueue

Thank you!!

Re: Remove logs from specific sourcetype being indexed

newsplunker1 — Mon, 13 Feb 2023 22:33:32 GMT

Sorry I meant sourcetype NOT index

Re: Remove logs from specific sourcetype being indexed

gcusello — Tue, 14 Feb 2023 08:24:05 GMT

Hi @newsplunker1,

using this configuration, you discard all logs with that sourcetype, so the question is: why don't you disable the related input instead log filtering after input?

the solution you describe is useful to discard selected logs not all logs.

To discard all logs from a predefined sourcetype it's easier to disable input for that sourcetype.

Ciao.

Giuseppe

Re: How to remove logs from specific sourcetype being indexed?

scelikok — Tue, 14 Feb 2023 08:57:16 GMT

Hi @newsplunker1,

Palo Alto TA uses sourcetype renaming. You should apply your transform to original sourcetype which is pan:log or pan_log. But as @gcusellowarned, above setting will filter everything in that case. Your regex must be specific. Please try below;

props.conf [pan:log] TRANSFORMS-filter_pan_userid = filter_pan_user_id transforms.conf [filter_pan_user_id] REGEX = ^[^,]+,[^,]+,[^,]+,USERID, DEST_KEY = queue FORMAT = nullQueue

Re: Remove logs from specific sourcetype being indexed

newsplunker1 — Tue, 14 Feb 2023 15:11:49 GMT

Hi @gcusello

Thanks for taking the time to look into this.

I could not find that sourcetype in the inputs.conf . Splunk monitors the following file

source="/var/log/splunk/network/paloalto/IP@/syslog.log"

my inputs.conf

[monitor:///var/log/splunk/network/paloalto/IP@/syslog.log]
disabled = 0
host =
host_segment = 6
sourcetype = pan:log
index = test

I can only see the pan:userid in the props conf

Re: How to remove logs from specific sourcetype being indexed?

newsplunker1 — Tue, 14 Feb 2023 15:14:27 GMT

Thanks @scelikok - I ll test it and report back

Re: Remove logs from specific sourcetype being indexed

gcusello — Tue, 14 Feb 2023 15:17:41 GMT

Hi @newsplunker1,

if you want to discard all logs with the sourcetype=pan:userid and you have only this input with this sourcetype, you can simply disable this input using "disabled = 1".

Ciao.

Giuseppe

Re: Remove logs from specific sourcetype being indexed

newsplunker1 — Tue, 14 Feb 2023 15:23:05 GMT

This will disable all sourcetypes

pan:threat

pan:traffic

pan:system

I just want to disable or ignore the logs for pan:userid sourcetype

Re: Remove logs from specific sourcetype being indexed

gcusello — Tue, 14 Feb 2023 15:27:37 GMT

Hi @newsplunker1,

as @scelikok said, this Add-on makes a transformation changing the sourcetype.

In this case the solution is the original one to install on Indexers or (if present) on Heavy Forwarders.

on props.conf:

[pan:log] TRANSFORMS-filter_pan_userid = filter_pan_user_id

transforms.conf:

[filter_pan_user_id] REGEX = ^[^,]+,[^,]+,[^,]+,USERID, DEST_KEY = queue FORMAT = nullQueue

Ciao.

Giuseppe