Getting Data In

Deploying blacklist configuration in inputs.conf to universal forwarder

jcbrendsel
Path Finder

I am having problems blacklisting a sourcefile from being indexed.

We currently run version 4.3 and deploy configurations to a number of remote universal forwarders.

By default, our universal forwarder indexes index everything, defined as follows:

#/opt/splunkforwarder/etc/apps/search/local/inputs.conf
[monitor:///var/log]
disabled = false

There are several logs that are unnecessary and that generate large logs that I would like to stop from getting indexed. To do that, I modified inputs.conf on the indexer/search head as follows:

#/opt/splunk/etc/deployment-apps/forwarder/local/inputs.conf
[monitor:///var/log/httpd]
blacklist = goonhilly_access

As you can see, this inputs.conf file is different than the one that is used for the default configuration (search).

I then deployed this to the remote universal forwarder in question and restarted.

The problem is that the file is still getting indexed.

Do I have a problem with inputs.conf files being in conflict?

The
Any ideas?

0 Karma
1 Solution

dwaddle
SplunkTrust
SplunkTrust

By default, monitor:// stanzas look for files recursively and settings in one stanza do not affect the other. So, yes, it's entirely possible that your /var/log/ stanza is ignoring your blacklist for /var/log/httpd. A couple of workable options include:

  1. Blacklist "/var/log/httpd/.*" in your /var/log stanza and use the /var/log/httpd stanza to get all of those.
  2. Blacklist goonhilly_access in your /var/log stanza and don't have a /var/log/httpd stanza at all.

There are CLI commands and REST endpoints to tell you which stanzas are detecting which files. One of the most useful ways of display this is with Amrit's script @ http://blogs.splunk.com/2011/01/02/did-i-miss-christmas-2/

View solution in original post

Michael
Contributor

I believe the line needs to be in regex format (according to: http://docs.splunk.com/Documentation/Splunk/6.3.0/Data/Whitelistorblacklistspecificincomingdata).

so, it should read:
goonhilly_access$

where $ = end of line

0 Karma

dwaddle
SplunkTrust
SplunkTrust

By default, monitor:// stanzas look for files recursively and settings in one stanza do not affect the other. So, yes, it's entirely possible that your /var/log/ stanza is ignoring your blacklist for /var/log/httpd. A couple of workable options include:

  1. Blacklist "/var/log/httpd/.*" in your /var/log stanza and use the /var/log/httpd stanza to get all of those.
  2. Blacklist goonhilly_access in your /var/log stanza and don't have a /var/log/httpd stanza at all.

There are CLI commands and REST endpoints to tell you which stanzas are detecting which files. One of the most useful ways of display this is with Amrit's script @ http://blogs.splunk.com/2011/01/02/did-i-miss-christmas-2/

jcbrendsel
Path Finder

That did the trick. Thanks.

0 Karma
Get Updates on the Splunk Community!

Splunk Education - Fast Start Program!

Welcome to Splunk Education! Splunk training programs are designed to enable you to get started quickly and ...

Five Subtly Different Ways of Adding Manual Instrumentation in Java

You can find the code of this example on GitHub here. Please feel free to star the repository to keep in ...

New Splunk APM Enhancements Help Troubleshoot Your MySQL and NoSQL Databases Faster

Splunk Observability has two new enhancements to make it quicker and easier to troubleshoot slow or frequently ...