Getting Data In

Deploying blacklist configuration in inputs.conf to universal forwarder

jcbrendsel
Path Finder

I am having problems blacklisting a sourcefile from being indexed.

We currently run version 4.3 and deploy configurations to a number of remote universal forwarders.

By default, our universal forwarder indexes index everything, defined as follows:

#/opt/splunkforwarder/etc/apps/search/local/inputs.conf
[monitor:///var/log]
disabled = false

There are several logs that are unnecessary and that generate large logs that I would like to stop from getting indexed. To do that, I modified inputs.conf on the indexer/search head as follows:

#/opt/splunk/etc/deployment-apps/forwarder/local/inputs.conf
[monitor:///var/log/httpd]
blacklist = goonhilly_access

As you can see, this inputs.conf file is different than the one that is used for the default configuration (search).

I then deployed this to the remote universal forwarder in question and restarted.

The problem is that the file is still getting indexed.

Do I have a problem with inputs.conf files being in conflict?

The
Any ideas?

0 Karma
1 Solution

dwaddle
SplunkTrust
SplunkTrust

By default, monitor:// stanzas look for files recursively and settings in one stanza do not affect the other. So, yes, it's entirely possible that your /var/log/ stanza is ignoring your blacklist for /var/log/httpd. A couple of workable options include:

  1. Blacklist "/var/log/httpd/.*" in your /var/log stanza and use the /var/log/httpd stanza to get all of those.
  2. Blacklist goonhilly_access in your /var/log stanza and don't have a /var/log/httpd stanza at all.

There are CLI commands and REST endpoints to tell you which stanzas are detecting which files. One of the most useful ways of display this is with Amrit's script @ http://blogs.splunk.com/2011/01/02/did-i-miss-christmas-2/

View solution in original post

Michael
Contributor

I believe the line needs to be in regex format (according to: http://docs.splunk.com/Documentation/Splunk/6.3.0/Data/Whitelistorblacklistspecificincomingdata).

so, it should read:
goonhilly_access$

where $ = end of line

0 Karma

dwaddle
SplunkTrust
SplunkTrust

By default, monitor:// stanzas look for files recursively and settings in one stanza do not affect the other. So, yes, it's entirely possible that your /var/log/ stanza is ignoring your blacklist for /var/log/httpd. A couple of workable options include:

  1. Blacklist "/var/log/httpd/.*" in your /var/log stanza and use the /var/log/httpd stanza to get all of those.
  2. Blacklist goonhilly_access in your /var/log stanza and don't have a /var/log/httpd stanza at all.

There are CLI commands and REST endpoints to tell you which stanzas are detecting which files. One of the most useful ways of display this is with Amrit's script @ http://blogs.splunk.com/2011/01/02/did-i-miss-christmas-2/

View solution in original post

jcbrendsel
Path Finder

That did the trick. Thanks.

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!