topic Re: Monitoring Forward activity in Getting Data In

Monitoring Forward activity

ruiaires — Thu, 20 Jan 2011 21:32:48 GMT

I have a Splunk server that receives data from 2 normal (not light) forwarders.

In the forwarders, I had to create a local index with the same name as the server and I'm using local file monitors.

Everything is OK and data is getting to the server as it should.

The problem is, I can't seem to monitor the activity on the forwarders: 1. there are no events in the _internal indexes 2. the default activity searches return no data 3. in the main server, I also cannot find _internal index data related to the forwarder's activity

A few days ago, one of the forwaders had splunkd using > 10% of CPU and I have no way of knowing or diagnosing the problem.

So, I should I monitor the forwarders' activity ?

Re: Monitoring Forward activity

tedder — Fri, 21 Jan 2011 00:57:59 GMT

Forwarders don't forward their _internal activity by default. Add the following to your inputs.conf:

[monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log]
_TCP_ROUTING = * 
index = _internal

Here's a conversation about it.

Re: Monitoring Forward activity

Simeon — Sat, 22 Jan 2011 02:07:10 GMT

You can utilize the settings in the LightForwarder app that send the internal logs onto the indexer. However, this only changes where the logs are located. To do this, you must edit the inputs.conf on your forwarder to reflect the following:

[monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log]
_TCP_ROUTING = *
index = _internal

[monitor://$SPLUNK_HOME/var/log/splunk/metrics.log]
_TCP_ROUTING = *
index = _internal

10% utilization is not uncommon for a full forwarder if it is doing some heavy processing. Without more detail of your inputs, this will be hard to diagnose.