Can't get a blacklist to work - please help!

[Branden]

I'm trying to get a blacklist in my inputs.conf to work correctly, but it's just not happening. It seems so simple... I am monitoring /logs.

Here's what I need:

• I need all files with a .bz2 and .gz suffix blacklisted
• I need all subfolders that start with esx-abcd- (and all files and folders under it) to be blacklisted. For example, /logs/esx-abcd-01/2011/09/14/syslog etc....

I tried this, and variants of it, but it didn't work:

blacklist = (^esx-abcd-|\.bz2$|\.gz$)

It seems like no matter what I try, it keeps indexing the directories under esx-abdc-*. I've even tried hard-coding in specific directory names (for testing purposes), yet it still goes ahead and indexes everything under it. Something obviously isn't right.

Can someone help?

Thanks!

[Branden]

I'll be honest, I have to give tech support credit for this one.... I kick myself because I was so close at one point:

blacklist = ^esx-tssdc-.*|\.(gz|gz2)$

[jrwilk01]

Try this:

blacklist = (?i:esx-abcd|\.bz2$|\.gz$)

That will make it a case-insensitive regex, and remove the anchor just in case there is a path element prepended.

Also, make sure that your monitor line doesn't have a trailing / on the path. I'm not sure it would matter, but just to be safe.

[Branden]

Thank you both for your comments!

[sdwilkerson]

Branden,

It is hard to say exactly, since you didn't include the rest of the stanza and we don't know what your monitor line looks like, but generally Splunk pulls the entire PATH of the files it monitors (this typically shows up as the "source" field. Therefore, your regex, "^esx-*" assumes that the beginning of the line starts with "esx". If this is windows, the beginning of the line is probably something like C:... and if it is Linux it might be something like "/var/log". It is unlikely the path to the monitored file will begin with "esx" without something in front of it.

Try removing the ^ since that might match, or stating the explicit path that should appear in appropriate regex format.

Best,
Sean

[Branden]

Sean,
Thank you for your response. It's a linux system, and I tried with and without the ^. Nothing seems to work...
The Splunk docs in http://docs.splunk.com/Documentation/Splunk/latest/Data/Whitelistorblacklistspecificincomingdata?r=s...
implied that the blacklist regex is relative to what is specified in the [monitor] stanza. They use this example:
" [monitor:///mnt/logs]
blacklist = (archive|historical|.bak$)

The above example tells Splunk to ignore all files under /mnt/logs/ within the archive or historical directories and all files ending in *.bak. "