topic Blacklist directory names not working in Getting Data In

Blacklist directory names not working

jackal242 — Fri, 03 Dec 2010 04:51:12 GMT

I've added the following blacklist line:

[monitor:///usr/local/alert/logs]
blacklist = (bak|sqlsync|syncdb_log|sql_bak|WebCorder)

And yet I see in my sources the following files are being indexed for that monitor:

/usr/local/alert/logs/bak/ipdlog_new.12133 | 17,016
/usr/local/alert/logs/bak/ipdlog_new.18100 | 15,770
/usr/local/alert/logs/bak/ipdlog_new.9881 | 15,727 
/usr/local/alert/logs/WebCorder/FF27774.65391.72094_001_act_src.html | 1
/usr/local/alert/logs/WebCorder/FF27774.65391.72094_002_act_src.html | 1

So clearly the blacklist is not working.

Any help?

Re: Blacklist directory names not working

jackal242 — Fri, 03 Dec 2010 05:26:45 GMT

I found the problem.

The problem is that I had already indexed the directory once without the blacklist and nothing removed those sources.

I thought removing the monitor and readding it with the blacklist would remove all the unwanted directories.

I had to do a "splunk clean eventdata" to get it to remove all the unwanted source directories that are now in the blacklist.

Re: Blacklist directory names not working

dwaddle — Fri, 03 Dec 2010 20:53:10 GMT

(Some additional information for you)

The "Search" app's initial Dashboard (the Summary page) is generated by running several searches against the already-indexed data. The sources list runs the following search:

| metadata sources

This includes an up to date event count which is updated roughly every 25 seconds. More info on the metadata command is available at http://www.splunk.com/base/Documentation/latest/SearchReference/Metadata.

As you commented, removing an input from Splunk does not delete already indexed data from the index - it stops Splunk from indexing any additional data from those files.

Re: Blacklist directory names not working

dwaddle — Fri, 03 Dec 2010 20:53:38 GMT

Thanks Amrit for the additional detail here.