topic Blacklisting clarification in Getting Data In

Blacklisting clarification

Voltaire — Tue, 20 Nov 2012 02:18:48 GMT

I am attempting to blacklist all files that end with these extensions in my inputs.conf file. The blacklist is not working correctly. These are image files located under a /images directory on the web server. These files are Not in the same directory as the log file. They are directory entries in the nginx-access log file contents.
For example

/wd/code/websites/wd-current/www/images/* 
/wd/code/websites/wd-current/www/js/resources/*
/wd/code/websites/sample.it/www/resources/*

I have attempted a few different methods. Any suggestions?

[default]
host = oh.br0ther.com

[monitor:///var/log/nginx-access.log]
blacklist= \.(jpg|png|gif|mov|js|swf|mp4|jar|signed|flv)$

Re: Blacklisting clarification

martin_mueller — Tue, 20 Nov 2012 08:09:53 GMT

Are you trying to block files from being read by Splunk or to block specific lines of a logfile from being indexed?

Re: Blacklisting clarification

Voltaire — Tue, 20 Nov 2012 18:27:02 GMT

I am trying to block files from being read by splunk in those directories.
Thank you.

Re: Blacklisting clarification

lguinn2 — Tue, 20 Nov 2012 18:45:59 GMT

Your stanza will monitor only the following files:

1 - files named /var/log/nginx-access.log

2 - files underneath a directory named /var/log/nginx-access.log

Since this is not the stanza that is monitoring the directories that you name, putting the blacklist here will not help.

I don't see anything wrong with your blacklist, it just needs to be moved so that it will be part of the proper monitor stanza.

Re: Blacklisting clarification

sdaniels — Tue, 20 Nov 2012 18:49:31 GMT

In your file monitoring stanza above you are referencing a file and not a directory. If I wanted to monitor the images directory and blacklist all of the image files I would so something like this:

[monitor:///wd/code/websites/wd-current/www/images/*]
index = myindex
sourcetype = mysourcetype
blacklist= \.(jpg|png|gif|mov|js|swf|mp4|jar|signed|flv)$

http://docs.splunk.com/Documentation/Splunk/5.0/Data/Monitorfilesanddirectories

Re: Blacklisting clarification

Voltaire — Tue, 20 Nov 2012 19:04:03 GMT

Thank you Lisa!

Re: Blacklisting clarification

Voltaire — Tue, 20 Nov 2012 19:07:20 GMT

Excellent, just modified inputs.conf on my forwarder. Would this command work if I wanted to exclude all files under the www directory in any of the subfolders? or do I have to add a different syntax ?
for example
[monitor:///wd/code/websites/wd-current/www/.../*]

Thank you!

Re: Blacklisting clarification

sdaniels — Tue, 20 Nov 2012 19:09:33 GMT

Some examples here. You can use elipsis wildcards or *.

http://docs.splunk.com/Documentation/Splunk/5.0/data/Specifyinputpathswithwildcards

Re: Blacklisting clarification

Voltaire — Wed, 21 Nov 2012 01:22:06 GMT

Nice ! thats where I found the reference. Thank you!