topic inputs.conf whitelist blacklist question in Getting Data In

inputs.conf whitelist blacklist question

ebailey — Mon, 28 Sep 2020 16:31:24 GMT

Greetings

I have trying to gather logs by sifting through three levels of the file system with a white list and blacklist. It is not working and outside of creating a very long list of monitors I am not sure what to do anymore.

Any help is much appreciated.

Thanks!

My config

[monitor:///opt/transient/prod/data/xxxxxx/xxx/xx_ci/log/]
recursive = true
sourcetype=prd_xxx_xx_log
disabled = false
crcSalt =
whitelist = .dat$|.log$|.err$|.txt$
blacklist = .gz$

[monitor:///opt/transient/prod/data/xxxxxx/xxx/xxx_ci/error/]
recursive = true
sourcetype=prd_xxx_xx_error
disabled = false
crcSalt =
whitelist = .dat$|.log$|.err$|.txt$
blacklist = .gz$

Any ideas?

Re: inputs.conf whitelist blacklist question

lguinn2 — Fri, 02 May 2014 16:22:44 GMT

Yes! Ditch the blacklist. You only need the whitelist. You also don't need the recursive - that's the default.

Finally - why do you have the crcSalt?

Re: inputs.conf whitelist blacklist question

ebailey — Mon, 05 May 2014 11:10:21 GMT

Thanks!

I use crcSalt with the expectation it is going to be needed. I pulled it out.

Question - I made the changes you recommended and the results look good except now I am picking up files such as

test.dat.lock also I am picking up all . files such as

.work

Any ideas?

Thanks

Re: inputs.conf whitelist blacklist question

neelamssantosh — Mon, 28 Sep 2020 17:35:32 GMT

why to give 'blacklist' of Specific extensions of compressed files to exclude, where splunk already ignores..

packed_extensions_list:
bz, bz2, tbz, tbz2, Z, gz, tgz, tar, zip

Re: inputs.conf whitelist blacklist question

MuS — Tue, 16 Sep 2014 11:04:02 GMT

hmmm, this packed_extensions_list is an option to crawl.conf only http://docs.splunk.com/Documentation/Splunk/6.1.3/admin/Crawlconf

and here would be the correct statement regarding compressed files from http://docs.splunk.com/Documentation/Splunk/6.1.3/Data/MonitorFilesandDirectories

Splunk Enterprise decompresses archive files before it indexes them. It can handle these common archive file types: tar, gz, bz2, tar.gz, tgz, tbz, bz2, zip, and z.