topic Re: Is there a way to use whitelist or blacklist within linux log files? in Getting Data In

Is there a way to use whitelist or blacklist within linux log files?

dieguiariel — Fri, 02 Jun 2023 06:21:43 GMT

Hi! from the documentation

https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/Whitelistorblacklistspecificincomingdata

the whitelist and blacklist option only works with the filenames of logs.

Is there an option for data within the log file?

eg:

from this extract fo /var/log/messages:

May 28 18:00:01 xxxxxxxxxx kernel: type=1110 audit(1685311201.838:180500): pid=19649 uid=0 auid=0 ses=24140 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:setcred grantors=pam_env,pam_faillock,pam_localuser,pam_unix acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
May 28 18:00:01 xxxxxxxxxx CROND[19681]: (root) CMD (/usr/lib64/sa/sa1 1 1)
May 28 18:00:01 xxxxxxxxxx kernel: type=1104 audit(1685311201.905:180501): pid=19649 uid=0 auid=0 ses=24140 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:setcred grantors=pam_env,pam_faillock,pam_localuser,pam_unix acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
May 28 18:00:01 xxxxxxxxxx kernel: type=1106 audit(1685311201.941:180502): pid=19649 uid=0 auid=0 ses=24140 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:session_close grantors=pam_loginuid,pam_keyinit,pam_limits,pam_systemd acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
May 28 18:00:01 svr-spl-mat-01 systemd: Removed slice User Slice of root.
May 28 18:00:02 svr-spl-mat-01 snmpd[1359]: Connection from UDP: [ xxxxxxxxxx]:50765->[ xxxxxxxxxx]:161
May 28 18:00:02 svr-spl-mat-01 snmpd[1359]: Connection from UDP: [ xxxxxxxxxx]:50765->[ xxxxxxxxxx]:161
May 28 18:00:02 svr-spl-mat-01 snmpd[1359]: Connection from UDP: [10.138.211.15]:50765->[ xxxxxxxxxx]:161

i will like to blacklist all the snmpd events.

the file used is just an example, the real file is from an application but with sensitive data that i dont want to get into splunk.

Regards.

Re: Is there a way to use whitelist or blacklist within linux log files?

gcusello — Fri, 02 Jun 2023 07:36:17 GMT

Hi @dieguiariel,

it's possible to filter events on the Universal Forwarder only for windows events.

For all other events (as Unix events), you can filter events on the Indexers or (when present) on intermediate Heavy Forwarders.

The documentation to do this is at https://docs.splunk.com/Documentation/SplunkCloud/latest/Forwarding/Routeandfilterdatad#Filter_event_data_and_send_to_queues

in few words, you have to insert:

in props.conf

[source::/var/log/messages] TRANSFORMS-null= setnull

in transforms.conf

[setnull] REGEX = snmpd DEST_KEY = queue FORMAT = nullQueue

Ciao.

Giuseppe

Re: Is there a way to use whitelist or blacklist within linux log files?

dieguiariel — Fri, 02 Jun 2023 13:20:51 GMT

Hi Giuseppe, thanks one more cuestion, reading the doc it says:

You can eliminate unwanted data by routing it to the nullQueue, the Splunk equivalent of the Unix /dev/null device. When you filter out data in this way, the data is not forwarded and doesn't count toward your indexing volume.

so, when applied this on the indexer, has impact on the daily license?, lets say that the events filtered out are around 2gb daily, i'll save 2 gb?

Regards.

Re: Is there a way to use whitelist or blacklist within linux log files?

dieguiariel — Mon, 05 Jun 2023 12:47:22 GMT

tried the solution and works perfectly. Also is saving license.

The files were created on:

$SplunkHOME/etc/system/local/

Thanks!

Re: Is there a way to use whitelist or blacklist within linux log files?

gcusello — Mon, 05 Jun 2023 13:08:57 GMT

Hi @dieguiariel ,

yes, the nullQueue is the Splunk equivalent of the Unix /dev/null device.

in Splunk you consume license only for the indexed logs, if you filter a data source before indexing, you don't consume license for the deleted logs.

Obviously you cannot use these filtered logs.

Ciao.

Giuseppe