topic How do I configure Universal Forwarder to not send INFO Metrics over TCP? in Getting Data In

How do I configure Universal Forwarder to not send INFO Metrics over TCP?

markdixon — Fri, 06 Nov 2015 11:12:57 GMT

My ouputs conf looks like this:

[tcpout]
defaultgroup = logstash
disabled = false

forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.blacklist = (_audit|_internal|_introspection)

[tcpout:logstash]
server=localhost:7777
sendCookedData = false
useACK = true

As well as my actual events, I'm seeing loads of messages being emitted like this:

 INFO  Metrics - group=thruput, name=uncooked_output, instantaneous_kbps=0.176377, instantaneous_eps=0.096773, average_kbps=0.355449, total_k_processed=44.000000, kb=5.467773, ev=3.000000
 INFO  Metrics - group=thruput, name=thruput, instantaneous_kbps=0.176377, instantaneous_eps=0.096773, average_kbps=0.371606, total_k_processed=46.000000, kb=5.467773, ev=3.000000, load_average=0.030000
 INFO  Metrics - group=thruput, name=cooked_output, instantaneous_kbps=0.000000, instantaneous_eps=0.000000, average_kbps=0.000000, total_k_processed=0.000000, kb=0.000000, ev=0.000000
 INFO  Metrics - group=tcpout_connections, name=logstash:127.0.0.1:7777:0, sourcePort=8090, destIp=127.0.0.1, destPort=7777, _tcp_Bps=186.73, _tcp_KBps=0.18, _tcp_avg_thruput=0.39, _tcp_Kprocessed=46, _tcp_eps=0.10, kb=5.47

How can I eliminate these from the forwarder output?

Re: How do I configure Universal Forwarder to not send INFO Metrics over TCP?

lguinn2 — Mon, 09 Nov 2015 07:18:22 GMT

Splunk automatically forwards its internal logs. The inputs.conf settings can be disabled to stop this. The settings may be found in several places, but usually they are set in $SPLUNK_HOMEetc/apps/SplunkUniversalForwarder/default/inputs.conf

Since you shouldn't edit anything in a default directory, create a local directory and create an inputs.conf that contains

[monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log]
disabled = true

[monitor://$SPLUNK_HOME/var/log/splunk/metrics.log]
disabled = true

Do the same for $SPLUNK_HOME/etc/system/local/inputs.conf

[monitor://$SPLUNK_HOME/var/log/splunk]
disabled=true

If the problem continues, or you see other files in the tcp output stream, check all the inputs.conf files on your system. There may be a few other default inputs that you need to disable.

Re: How do I configure Universal Forwarder to not send INFO Metrics over TCP?

alexsayegh — Sun, 04 Feb 2018 14:29:38 GMT

Hi there,
I know this is old and all, but is it still valid on version 7.0.1?
Adding the file:

$SPLUNK_HOME/etc/apps/SplunkUniversalForwarder/local/inputs.conf

With content:

[monitor://$SPLUNK_HOME/var/log/splunk/metrics.log]
#_TCP_ROUTING = *
#index = _internal
disabled = true

Doesn't disable the metrics input.
(Trying to disable it since splunk-optimize goes crazy when trying to run on _internal index and ends up crashing the server out of memory).

Re: How do I configure Universal Forwarder to not send INFO Metrics over TCP?

alexsayegh — Sun, 04 Feb 2018 14:30:16 GMT

Editing the default/inputs.conf also doesn't

Re: How do I configure Universal Forwarder to not send INFO Metrics over TCP?

lguinn2 — Mon, 05 Feb 2018 19:39:33 GMT

Never edit the files in the default directories. Even if it works, your changes will be overwritten when you update Splunk. The files in the corresponding local directories always override the default directories.

This should still work in Splunk 7, but you are in the wrong directory. Do the same thing, but put it in $SPLUNK_HOME/etc/system/local (which is probably /opt/splunkforwarder/etc/system/local on a Linux box).

Re: How do I configure Universal Forwarder to not send INFO Metrics over TCP?

lguinn2 — Mon, 05 Feb 2018 20:04:38 GMT

New answer: what if you want to send some information to Splunk, but not everything?
Maybe you don't want the metrics, but you would like the errors, etc. from the splunkd.log

In $SPLUNK/HOME/etc/system/local/inputs.conf, only disable the metrics log

 [monitor://$SPLUNK_HOME/var/log/splunk/metrics.log]
 disabled = true

You can set the "log levels" on the forwarder by copying $SPLUNK_HOME/etc/log.cfg to $SPLUNK_HOME/etc/log-local.cfg
Edit $SPLUNK_HOME/etc/log-local.cfg to customize the logging, but remember that these logs are a primary source for Splunk Monitoring Console. These edits will mostly affect the splunkd.log
There are many log channels, and you don't need to reset all of them. Just change "INFO" to "WARN" on any categories where you want to reduce the messages. You can delete any lines that you want to leave at INFO level. The following channels should always be left at INFO level:

category.TailingProcessor=INFO
category.loader=INFO

Re: How do I configure Universal Forwarder to not send INFO Metrics over TCP?

alexsayegh — Tue, 06 Feb 2018 08:29:40 GMT

Thanks Iguinn, I know about the default directory, and I'll definitely try the log levels.
I migrated the whole indexer to a new Cloud instance, so there is no longer an issue with the tsids...but i'll test it out anyway.
Appreciate it

Re: How do I configure Universal Forwarder to not send INFO Metrics over TCP?

law — Fri, 06 Aug 2021 11:23:48 GMT

[monitor://$SPLUNK_HOME/var/log/splunk]
blacklist = metrics\.log

Re: How do I configure Universal Forwarder to not send INFO Metrics over TCP?

dems2234 — Thu, 21 Nov 2024 17:16:14 GMT

[monitor://$SPLUNK_HOME/var/log/splunk]
blacklist = metrics\.log$

metrics\.log$ is the correct regex assigned to the blacklist variable. It is possible the one provided won't work, or at least, it didn't work for me.