Is there a way to use whitelist or blacklist within linux log files?

[dieguiariel]

Hi! from the documentation 

https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/Whitelistorblacklistspecificincomingda...

the whitelist and blacklist option only works with the filenames of logs.

Is there an option for data within the log file?

eg:

from this extract fo /var/log/messages:

May 28 18:00:01 xxxxxxxxxx kernel: type=1110 audit(1685311201.838:180500): pid=19649 uid=0 auid=0 ses=24140 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:setcred grantors=pam_env,pam_faillock,pam_localuser,pam_unix acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
May 28 18:00:01 xxxxxxxxxx CROND[19681]: (root) CMD (/usr/lib64/sa/sa1 1 1)
May 28 18:00:01 xxxxxxxxxx kernel: type=1104 audit(1685311201.905:180501): pid=19649 uid=0 auid=0 ses=24140 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:setcred grantors=pam_env,pam_faillock,pam_localuser,pam_unix acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
May 28 18:00:01 xxxxxxxxxx kernel: type=1106 audit(1685311201.941:180502): pid=19649 uid=0 auid=0 ses=24140 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:session_close grantors=pam_loginuid,pam_keyinit,pam_limits,pam_systemd acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
May 28 18:00:01 svr-spl-mat-01 systemd: Removed slice User Slice of root.
May 28 18:00:02 svr-spl-mat-01 snmpd[1359]: Connection from UDP: [ xxxxxxxxxx]:50765->[ xxxxxxxxxx]:161
May 28 18:00:02 svr-spl-mat-01 snmpd[1359]: Connection from UDP: [ xxxxxxxxxx]:50765->[ xxxxxxxxxx]:161
May 28 18:00:02 svr-spl-mat-01 snmpd[1359]: Connection from UDP: [10.138.211.15]:50765->[ xxxxxxxxxx]:161

i will like to blacklist all the snmpd events.

the file used is just an example, the real file is from an application but with sensitive data that i dont want to get into splunk.

Regards.

[gcusello · Accepted solution]

Hi @dieguiariel,

it's possible to filter events on the Universal Forwarder only for windows events.

For all other events (as Unix events), you can filter events on the Indexers or (when present) on intermediate Heavy Forwarders.

The documentation to do this is at https://docs.splunk.com/Documentation/SplunkCloud/latest/Forwarding/Routeandfilterdatad#Filter_event...

in few words, you have to insert:

in props.conf

[source::/var/log/messages]
TRANSFORMS-null= setnull

in transforms.conf

[setnull]
REGEX = snmpd
DEST_KEY = queue
FORMAT = nullQueue

Ciao.

Giuseppe

[dieguiariel]

Hi Giuseppe, thanks one more cuestion,  reading the doc it says:

You can eliminate unwanted data by routing it to the nullQueue, the Splunk equivalent of the Unix /dev/null device. When you filter out data in this way, the data is not forwarded and doesn't count toward your indexing volume.

so, when applied this on the indexer, has impact on the daily license?, lets say that the events filtered out are around 2gb daily, i'll save 2 gb? 

Regards.

[gcusello]

Hi @dieguiariel ,

yes, the nullQueue is the Splunk equivalent of the Unix /dev/null device.

in Splunk you consume license only for the indexed logs, if you filter a data source before indexing, you don't consume license for the deleted logs.

Obviously you cannot use these filtered logs.

Ciao.

Giuseppe

[dieguiariel]

tried the solution and works perfectly. Also is saving license.

The files were created on:

$SplunkHOME/etc/system/local/

Thanks!