Getting Data In

Blacklisting clarification

Voltaire
Communicator

I am attempting to blacklist all files that end with these extensions in my inputs.conf file. The blacklist is not working correctly. These are image files located under a /images directory on the web server. These files are Not in the same directory as the log file. They are directory entries in the nginx-access log file contents.
For example

/wd/code/websites/wd-current/www/images/* 
/wd/code/websites/wd-current/www/js/resources/*
/wd/code/websites/sample.it/www/resources/*

I have attempted a few different methods. Any suggestions?

[default]
host = oh.br0ther.com

[monitor:///var/log/nginx-access.log]
blacklist= \.(jpg|png|gif|mov|js|swf|mp4|jar|signed|flv)$
Tags (1)
0 Karma

sdaniels
Splunk Employee
Splunk Employee

In your file monitoring stanza above you are referencing a file and not a directory. If I wanted to monitor the images directory and blacklist all of the image files I would so something like this:

[monitor:///wd/code/websites/wd-current/www/images/*]
index = myindex
sourcetype = mysourcetype
blacklist= \.(jpg|png|gif|mov|js|swf|mp4|jar|signed|flv)$

http://docs.splunk.com/Documentation/Splunk/5.0/Data/Monitorfilesanddirectories

0 Karma

Voltaire
Communicator

Nice ! thats where I found the reference. Thank you!

0 Karma

sdaniels
Splunk Employee
Splunk Employee

Some examples here. You can use elipsis wildcards or *.

http://docs.splunk.com/Documentation/Splunk/5.0/data/Specifyinputpathswithwildcards

0 Karma

Voltaire
Communicator

Excellent, just modified inputs.conf on my forwarder. Would this command work if I wanted to exclude all files under the www directory in any of the subfolders? or do I have to add a different syntax ?
for example
[monitor:///wd/code/websites/wd-current/www/.../*]

Thank you!

0 Karma

lguinn2
Legend

Your stanza will monitor only the following files:

1 - files named /var/log/nginx-access.log

2 - files underneath a directory named /var/log/nginx-access.log

Since this is not the stanza that is monitoring the directories that you name, putting the blacklist here will not help.

I don't see anything wrong with your blacklist, it just needs to be moved so that it will be part of the proper monitor stanza.

Voltaire
Communicator

Thank you Lisa!

0 Karma

Voltaire
Communicator

I am trying to block files from being read by splunk in those directories.
Thank you.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Are you trying to block files from being read by Splunk or to block specific lines of a logfile from being indexed?

0 Karma
Get Updates on the Splunk Community!

Splunk Up Your Game: Why It's Time to Embrace Python 3.9+ and OpenSSL 3.0

Did you know that for Splunk Enterprise 9.4, Python 3.9 is the default interpreter? This shift is not just a ...

See your relevant APM services, dashboards, and alerts in one place with the updated ...

As a Splunk Observability user, you have a lot of data you have to manage, prioritize, and troubleshoot on a ...

Cultivate Your Career Growth with Fresh Splunk Training

Growth doesn’t just happen—it’s nurtured. Like tending a garden, developing your Splunk skills takes the right ...