I am rolling out the universal forwarders to my domain controllers. All was going well untill I started installing it on my 2008R2 domain controllers. The universla forwarder works fine on my 2k3 and 2008 boxes. On my 2008R2 servers the agent checks in but doews not send any events. It looks like it gets it config from the deployment server but then it can't connect. I found this in the splunkd.log: No connection could be made
04-12-2011 14:25:40.970 -0400 WARN DeployedApplication - Installing app: inputs_win_sec to location: D:\program files\splunk\etc\apps\inputs_win_sec 04-12-2011 14:25:41.048 -0400 INFO DeployedApplication - Checksum mismatch 0 <> 14022092945545768778 for app: outputs_win. It will be reloaded again from: 10.136.255.33:8090/services/streams/deployment?name=default:forwarder_win_sec:outputs_win 04-12-2011 14:25:41.048 -0400 INFO DeployedApplication - Remote repository has resolved to: 10.136.255.33:8090/services/streams/deployment?name=default:forwarder_win_sec:outputs_win 04-12-2011 14:25:41.142 -0400 WARN HTTPClient - Unable to parse status line: HTTP/1.1 200 04-12-2011 14:25:41.142 -0400 INFO DeployedApplication - Downloaded url: 10.136.255.33:8090/services/streams/deployment?name=default:forwarder_win_sec:outputs_win to file: D:\program files\splunk\var\run\forwarder_win_sec\outputs_win-1302272649.bundle 04-12-2011 14:25:41.142 -0400 WARN DeployedApplication - Installing app: outputs_win to location: D:\program files\splunk\etc\apps\outputs_win 04-12-2011 14:25:41.220 -0400 WARN DeploymentClient - Restarting Splunkd... 04-12-2011 14:25:47.819 -0400 WARN TcpOutputFd - Connect to 10.x.x.x:9997 failed. No connection could be made because the target machine actively refused it. 04-12-2011 14:25:47.819 -0400 ERROR TcpOutputFd - Connection to host=10.x.x.x:9997 failed 04-12-2011 14:25:48.006 -0400 INFO TailingProcessor - Could not send data to output queue (parsingQueue), retrying... 04-12-2011 14:26:17.818 -0400 WARN TcpOutputFd - Connect to 10.x.x.x:9997 failed. No connection could be made because the target machine actively refused it.
Sounds like a networking problem.
The forwarder is saying that it cannot open a socket to your 10.x.x.x host on 9997.
You can probably rapidly rule out the client behavior by trying to telnet to that port from those systems. Likely, you will get the same error.
Does anyone know what the problem was? We are experiencing the same problem. The UF checks in but after a few hours, even 24 hours but eventually stops sending.
I had something similar. To fix, I opened the Firewall settings on the Splunk Server and added the Splunk Receiver port (9911) and Splunk Admin Port (8000) to the allowed exceptions and all worked fine.
Changes were made to the indexer server. By default, outbound connections from the forwarder server do not normally need firewall changes. The indexer will need to have the scope for the forwarder as well as the ports in use.