How to tell a Splunk Universal Forwarder to not to monitor its own log files?

[VipulPathak]

Hello Everyone,

We are trying to monitor log files on a server using the Splunk universal forwarder. The logs directory (say /logs/app3/Oct2015) is being monitored by Splunk forwarder.

/opt/splunkforwarder/etc/system/local/inputs.conf :
Reference: http://docs.splunk.com/Documentation/Splunk/6.3.0/Data/Monitorfilesanddirectorieswithinputs.conf

[monitor:///logs/app3/Oct2015/]
    sourcetype = access_common
    ignoreOlderThan = 7d
    recurse = true
    _TCP_ROUTING = toLogAggregator
    blacklist = /opt/splunkforwarder/...

[blacklist:/opt/splunkforwarder/...]

This is my Output configuration -

/opt/splunkforwarder/etc/system/local/outputs.conf :

Reference: http://docs.splunk.com/Documentation/Splunk/6.3.0/Forwarding/Configureforwarderswithoutputs.conf

[tcpout]
defaultGroup = toBeIgnored

# This target group will redirect events to Log aggregator listening on TCP socket.
[tcpout:toLogAggregator]
server = 10.20.176.207:9997
sendCookedData = false

[tcpout:toBeIgnored]

When I run the Splunk Universal Forwarder (v6.3), I am successfully able to monitor my logs directory, but Splunk is also sending me its own logs. I added a blacklist stanza to stop Splunk from monitoring and sending me its own logs, but no luck. My aggregator server is continuously receiving Splunk logs mixed with logs I am monitoring.

I added a TCP output group to drain events by default and explicitly marked my monitored directory to route logs using a different TCP group, but that also didn't work. Still getting bombarded with Splunk logs.

Does anyone knows, how to tell Splunk Forwarder to not to monitor itself but only monitor the directory I have requested for ?

regards,

-Vipul;

[pickerin · Accepted solution]

Do you happen to be running the "Splunk Add-on for Unix and Linux" TA on the forwarder?

You may have to disable various monitor stanzas in that application's inputs.conf.

[VipulPathak]

Thanks for the pointer pickerin.

I don't remember installing it, but since it is a shared environment with more than one admin users, I would definitely want to check for it. Where can I check for evidence of such plugin in action ? Any directory/config file etc. ? I checked the apps folder under $SPLUNK_HOME/etc but didn't find any folder by name matching/similar to "Splunk Add-on for Unix and Linux".

[lguinn2]

The add-ons will not forward internal Splunk logs...

[pickerin]

@VipulPathak - the add-on will be in the $SPLUNK_HOME/etc/apps/Splunk_TA_nix if installed.

@Iguinn - that's actually not true, the add-on, by default, has a monitor for /var/log, so it'll forward everything inside of that directory, which are "internal" logs to the Forwarder itself (though not internal logs to the Splunk process). The stanza is:

[monitor:///var/log]
whitelist=(\.log|log$|messages|secure|auth|mesg$|cron$|acpid$|\.out)
blacklist=(lastlog|anaconda\.syslog)
index=os
disabled = 0

[lguinn2]

Splunk does not store its internal log files in /var/log - it stores them in $SPLUNK_HOME/var/log/splunk

Which on a UF installed on Linux would usually be

/opt/splunkforwarder/var/log/splunk

[VipulPathak]

I didn't find "Splunk Add-on for Unix and Linux" on my installation, nor was it installed explicitly by any of the Admin users.

However, there was another plugin present in /opt/splunkforwarder/etc/apps/ by default, called "SplunkUniversalForwarder". Inside the plugin directory, there is a default directory that contains an inputs.conf file. This file contains the monitor statements for $SPLUNK_HOME/var/log folder.

################################
# Make sure these get forwarded
################################

[monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log]
disabled = true
_TCP_ROUTING = *
index = _internal

[monitor://$SPLUNK_HOME/var/log/splunk/metrics.log]
disabled = true
_TCP_ROUTING = *
index = _internal

I disabled them and Bingo, Splunk stopped flooding the destination with Splunk logs, while sending what I asked Splunk to forward.

Pretty Nice, problem solved. Thanks Pickerin and Lguinn for taking time to discuss and helping me pin point the problem.

However, it is really strange to see that any plugin can change the overall behavior of Splunk as a whole. Wouldn't that be cool, if a changes in the DEFAULT of any plugins, only affect that Plugin and not all applications as a whole ?

Thanks for the help.

-Vipul.

[lguinn2]

First, why don't you want Splunk to send its internal logs? They are not charged against your license and the indexer will only keep about 500GB of the data at most. They can be very useful when trying to monitor or debug your forwarder(s).

Second, I don't know why your blacklist stanza doesn't work. I have never used one. The blacklist within your [monitor:///logs/app3/Oct2015/] stanza definitely won't work.

The setting on the forwarder that sends the internal logs is stored in SPLUNK_HOME/etc/system/default/inputs.conf But don't edit anything in the default directory. Override it in the local directory.

To SPLUNK_HOME/etc/system/local/inputs.conf on the forwarder, add the following

[monitor://$SPLUNK_HOME/var/log/splunk]
disabled=true

[monitor://$SPLUNK_HOME/etc/splunk.version]
disabled=true

Get rid of the blacklists. Restart the forwarder for the new settings to take effect.

[VipulPathak]

Thanks Iguinn for the response,

Let me add more context here:
On the other side of the network connection is a Java application reading from ServerSocket. We are collecting logs from different servers and posting them to HDFS for further hadoop based analysis. The data we are collecting from application servers, is RAW (uncooked) data.

At this point, We are only interested in the application logs that is being watched.

I added the [monitor:///opt/splunkforwarder/var/log/splunk/] and disabled=true under it in the local/inputs.conf file, but there is absolutely no effect on the behavior.

I also tried to add multiple [monitor:] with listing each file under var/log/splunk in it, but still not effect.

Please suggest.

Thanks for your help.

[lguinn2]

I would add the stanzas exactly as I listed them. Not with the expanded path. Otherwise I am not sure that they will match for the override.

So you are saying that you are using a Universal Forwarder, but it isn't forwarding to a Splunk indexer! Interesting.

[VipulPathak]

No, there is not Splunk indexer in the picture. I am forwarding to my own Java application that reads on a Socket and post the data to HDFS. So this is similar to forwarding to 3rd party system.

(Splunk Forwarder) ====> (Java App
Reading on Socket) =====> (Hadoop's
HDFS)

Thats the reason, my output configuration is setting the attribute: sendCookedData = false.

[lguinn2]

Cool. I get it.

[VipulPathak]

Yeah, except that Splunk Universal Forwarder is continuing to send metrics and logs to the destination, mixing with other logs and I can't stop it 🙂

Thanks for your support though 🙂

[lguinn2]

File a support ticket: http://www.splunk.com/en_us/support-and-services.html

This should have worked. Disabling the stanzas in etc/system/local/inputs.conf should have worked. As I understand it, the original blacklist stanza that you used should have worked.

As a last option - edit the etc/system/defaults/inputs.conf file on the forwarder. Put "disabled=true" in each stanza as shown above. Restart the forwarder. While you aren't supposed to edit default config files, this should also work. But even if it does, follow up with support about the other options.

[pj]

Having tried to disable monitoring of the metrics and splunkd logs today, it does look as though disabled = 1 doesnt work for these sources on the universal forwarder. There are monitors for these in both the splunk universal forwarder app on the splunk forwarder and in the system default. I put an entries in system local to disable all the them and checked btool after a restart to confirm they are disabled. However internal logs still get forwarded. I didnt try hardcoding into default, cos that is just a pain. I guess another option would be to nullqueue on the indexers. However it is frustrating that this doesnt appear to work in accordance with accepted Splunk rules.