Getting Data In

Blacklisting clarification

Communicator

I am attempting to blacklist all files that end with these extensions in my inputs.conf file. The blacklist is not working correctly. These are image files located under a /images directory on the web server. These files are Not in the same directory as the log file. They are directory entries in the nginx-access log file contents.
For example

/wd/code/websites/wd-current/www/images/* 
/wd/code/websites/wd-current/www/js/resources/*
/wd/code/websites/sample.it/www/resources/*

I have attempted a few different methods. Any suggestions?

[default]
host = oh.br0ther.com

[monitor:///var/log/nginx-access.log]
blacklist= \.(jpg|png|gif|mov|js|swf|mp4|jar|signed|flv)$
Tags (1)
0 Karma

Splunk Employee
Splunk Employee

In your file monitoring stanza above you are referencing a file and not a directory. If I wanted to monitor the images directory and blacklist all of the image files I would so something like this:

[monitor:///wd/code/websites/wd-current/www/images/*]
index = myindex
sourcetype = mysourcetype
blacklist= \.(jpg|png|gif|mov|js|swf|mp4|jar|signed|flv)$

http://docs.splunk.com/Documentation/Splunk/5.0/Data/Monitorfilesanddirectories

0 Karma

Communicator

Nice ! thats where I found the reference. Thank you!

0 Karma

Splunk Employee
Splunk Employee

Some examples here. You can use elipsis wildcards or *.

http://docs.splunk.com/Documentation/Splunk/5.0/data/Specifyinputpathswithwildcards

0 Karma

Communicator

Excellent, just modified inputs.conf on my forwarder. Would this command work if I wanted to exclude all files under the www directory in any of the subfolders? or do I have to add a different syntax ?
for example
[monitor:///wd/code/websites/wd-current/www/.../*]

Thank you!

0 Karma

Legend

Your stanza will monitor only the following files:

1 - files named /var/log/nginx-access.log

2 - files underneath a directory named /var/log/nginx-access.log

Since this is not the stanza that is monitoring the directories that you name, putting the blacklist here will not help.

I don't see anything wrong with your blacklist, it just needs to be moved so that it will be part of the proper monitor stanza.

Communicator

Thank you Lisa!

0 Karma

Communicator

I am trying to block files from being read by splunk in those directories.
Thank you.

0 Karma

SplunkTrust
SplunkTrust

Are you trying to block files from being read by Splunk or to block specific lines of a logfile from being indexed?

0 Karma