I've set up a forwarder but I'm not receving any events on the splunk indexer.

[Jaci]

I've verified that the indexer (receiver) is the same or later version of Splunk as the forwarder. What log or configuration files can I look at to troubleshoot this problem?

[Jaci · Accepted solution]

On the forwarder check: sysinfo.txt (verify general system info) outputs.conf (verify settings) metrics.log (search for tcpout_connections, destport=xxxx) splunkd.log (search for Error and WARN "failed to make connections")

Configuration/log files to check on indexer: inputs.conf (search for splunktcp:\xxxx) metrics.log (search for data coming from forwarder) splunkd.log (search for Error, tcpin_connections (look for forwarder hostname/IP))

Also, you can try running a search on the indexer to see if data came in from the forwarder.

[Simeon]

I would begin by confirming basic connectivity. I will assume we are on linux and using the default forwarding port of 9997 (no ssl):

Look for your receiving port to be open on the indexer:

> netstat -an | grep 9997

**This should return an active TCP listener on 9997

Look for your receiving port to be connected to from the forwarder:

> netstat -an | grep 9997 

**This should return an active TCP connection TO port 9997 on your indexer

If neither of the above are operational, then fowarding will not work. You should review if you have properly configured receiving and forwarding. Note that you may need to restart to enable forwarding.

Next, you should run a search to find the forwarder connection on the indexer:

index=_internal source=*metrics.log tcpin_connections

You should see an event very similar to below with your forwarder IP address:

04-23-2010 23:00:36.887 INFO Metrics - group=tcpin_connections, 192.1.168.10:36924:9997, connectionType=cooked, sourcePort=36924, sourceHost=forwarder.splunk.com, sourceIp=10.8.240.201, destPort=9997, _tcp_Bps=0.00, _tcp_KBps=0.00, _tcp_avg_thruput=2.08, _tcp_Kprocessed=10078.00, _tcp_eps=0.00

If you see positive values for tcp_Kprocessed, that means your forwarder is connected and has transferred data. If you do not see the above event in your metrics.log file (_internal index), you should then refer to the splunkd.log on your indexer and forwarder. Splunk will log an entry in the splunkd.log file when a forwarder has connected.

[jrodman]

Here's a starting point: http://www.splunk.com/wiki/Community:TroubleshootingForwarding

[BunnyHop · Accepted solution]

You can do the command "splunk list forward-server" to see if the forward-server is active on the forwarder. If it's inactive, it usually means you have not enabled the receiver to receive forwarded data.

Go to the receiver and then browse to the "Manager > Forwarding and receiving > under receive data select Receive data from forwarder. The port specified here should be the same port the forwarders are configured to send data. So if you're receiver is set to receive forwarded data to port 8889, then you should have this listed when you do the "splunk list forward-server" command:

splunkserver:8889

[chris]

If you are sending data to a specific index. It has to be created on the indexer first. You can see if an index contains data (and when the first and last events arrived) in the web gui.

[V_at_Splunk]

splunkd.log would be a good start; please check it on the forwarder first, then indexer; look for ERROR lines.

Of config files, outputs.conf on the forwarder is of interest.

Before all that, though, be sure to check network connectivity with ping(8) or ping.

[Jaci · Accepted solution]

On the forwarder check: sysinfo.txt (verify general system info) outputs.conf (verify settings) metrics.log (search for tcpout_connections, destport=xxxx) splunkd.log (search for Error and WARN "failed to make connections")

Configuration/log files to check on indexer: inputs.conf (search for splunktcp:\xxxx) metrics.log (search for data coming from forwarder) splunkd.log (search for Error, tcpin_connections (look for forwarder hostname/IP))

Also, you can try running a search on the indexer to see if data came in from the forwarder.