Getting Data In

inputs.conf whitelist blacklist question

Communicator

Greetings

I have trying to gather logs by sifting through three levels of the file system with a white list and blacklist. It is not working and outside of creating a very long list of monitors I am not sure what to do anymore.

Any help is much appreciated.

Thanks!

My config

[monitor:///opt/transient/prod/data/xxxxxx/xxx/xx_ci/log/]
recursive = true
sourcetype=prd_xxx_xx_log
disabled = false
crcSalt =
whitelist = .dat$|.log$|.err$|.txt$
blacklist = .gz$

[monitor:///opt/transient/prod/data/xxxxxx/xxx/xxx_ci/error/]
recursive = true
sourcetype=prd_xxx_xx_error
disabled = false
crcSalt =
whitelist = .dat$|.log$|.err$|.txt$
blacklist = .gz$

Any ideas?

0 Karma
1 Solution

Legend

Yes! Ditch the blacklist. You only need the whitelist. You also don't need the recursive - that's the default.

Finally - why do you have the crcSalt?

View solution in original post

0 Karma

Contributor

why to give 'blacklist' of Specific extensions of compressed files to exclude, where splunk already ignores..

packed_extensions_list:
bz, bz2, tbz, tbz2, Z, gz, tgz, tar, zip

0 Karma

SplunkTrust
SplunkTrust

hmmm, this packed_extensions_list is an option to crawl.conf only http://docs.splunk.com/Documentation/Splunk/6.1.3/admin/Crawlconf

and here would be the correct statement regarding compressed files from http://docs.splunk.com/Documentation/Splunk/6.1.3/Data/MonitorFilesandDirectories

  • Splunk Enterprise decompresses archive files before it indexes them. It can handle these common archive file types: tar, gz, bz2, tar.gz, tgz, tbz, bz2, zip, and z.

Legend

Yes! Ditch the blacklist. You only need the whitelist. You also don't need the recursive - that's the default.

Finally - why do you have the crcSalt?

View solution in original post

0 Karma

Communicator

Thanks!

I use crcSalt with the expectation it is going to be needed. I pulled it out.

Question - I made the changes you recommended and the results look good except now I am picking up files such as

test.dat.lock also I am picking up all . files such as

.work

Any ideas?

Thanks

Ed

0 Karma