Getting Data In
Highlighted

2008R2 universal forwarder issue

New Member

I am rolling out the universal forwarders to my domain controllers. All was going well untill I started installing it on my 2008R2 domain controllers. The universla forwarder works fine on my 2k3 and 2008 boxes. On my 2008R2 servers the agent checks in but doews not send any events. It looks like it gets it config from the deployment server but then it can't connect. I found this in the splunkd.log: No connection could be made

04-12-2011 14:25:40.970 -0400 WARN  DeployedApplication - Installing app: inputs_win_sec to location: D:\program files\splunk\etc\apps\inputs_win_sec
04-12-2011 14:25:41.048 -0400 INFO  DeployedApplication - Checksum mismatch 0 <> 14022092945545768778 for app: outputs_win.   It will be reloaded again from: 10.136.255.33:8090/services/streams/deployment?name=default:forwarder_win_sec:outputs_win
04-12-2011 14:25:41.048 -0400 INFO  DeployedApplication - Remote repository has resolved to:  10.136.255.33:8090/services/streams/deployment?name=default:forwarder_win_sec:outputs_win
04-12-2011 14:25:41.142 -0400 WARN  HTTPClient - Unable to parse status line: HTTP/1.1 200
04-12-2011 14:25:41.142 -0400 INFO  DeployedApplication - Downloaded url: 10.136.255.33:8090/services/streams/deployment?name=default:forwarder_win_sec:outputs_win to file: D:\program files\splunk\var\run\forwarder_win_sec\outputs_win-1302272649.bundle
04-12-2011 14:25:41.142 -0400 WARN  DeployedApplication - Installing app: outputs_win to location: D:\program files\splunk\etc\apps\outputs_win
04-12-2011 14:25:41.220 -0400 WARN  DeploymentClient - Restarting Splunkd...
04-12-2011 14:25:47.819 -0400 WARN  TcpOutputFd - Connect to 10.x.x.x:9997 failed. No connection could be made because the target machine actively refused it.
04-12-2011 14:25:47.819 -0400 ERROR TcpOutputFd - Connection to host=10.x.x.x:9997 failed
04-12-2011 14:25:48.006 -0400 INFO  TailingProcessor - Could not send data to output queue (parsingQueue), retrying...
04-12-2011 14:26:17.818 -0400 WARN  TcpOutputFd - Connect to 10.x.x.x:9997 failed. No connection could be made because the target machine actively refused it.
Tags (1)
0 Karma
Highlighted

Re: 2008R2 universal forwarder issue

Splunk Employee
Splunk Employee

Sounds like a networking problem.

The forwarder is saying that it cannot open a socket to your 10.x.x.x host on 9997.

Possibilities include:

  • Firewall (unlikely, connection refused is not how firewalls normally behave)
  • Routing: perhaps 10.x.x.x means something different on that network?
  • port 9997 is not open:
    • Maybe splunk is not running on that host right now
    • Maybe splunk is not configured to receive data on 9997 on that host

You can probably rapidly rule out the client behavior by trying to telnet to that port from those systems. Likely, you will get the same error.

0 Karma
Highlighted

Re: 2008R2 universal forwarder issue

Communicator

Does anyone know what the problem was? We are experiencing the same problem. The UF checks in but after a few hours, even 24 hours but eventually stops sending.

0 Karma
Highlighted

Re: 2008R2 universal forwarder issue

Path Finder

Same here, i am seeing this behavior on my 2008 systems

0 Karma
Highlighted

Re: 2008R2 universal forwarder issue

New Member

I had something similar. To fix, I opened the Firewall settings on the Splunk Server and added the Splunk Receiver port (9911) and Splunk Admin Port (8000) to the allowed exceptions and all worked fine.

0 Karma
Highlighted

Re: 2008R2 universal forwarder issue

Path Finder

Were your firewall changes done on the indexer or the universal forwarder?

0 Karma
Highlighted

Re: 2008R2 universal forwarder issue

New Member

Changes were made to the indexer server. By default, outbound connections from the forwarder server do not normally need firewall changes. The indexer will need to have the scope for the forwarder as well as the ports in use.

0 Karma