It looks like it is the outputs.conf that is the culprit. After removing outputs.conf from splunk\etc\apps\sendtoindexer\local folder, the problem disappeared after a reboot. I think additional instructions need to be appended to the page below as it instructs to place outputs.conf into the local box: http://docs.splunk.com/Documentation/MSApp/1.1.0/MSInfra/Createthesendtoindexerapp Thanks for all the help! ... View more

That's a typo. Sorry. I tried to telnet to port 9997 but still failed. I decided to reboot and re-apply the netsh advfirewall setting to disable the firewall on both servers. For the first 300 seconds, I can telnet to port 9997 of the server from the client. I still get the "Forwarding to indexer group default-autolb-group blocked for xxx seconds." message from the 100th second after reboot. After 300 seconds, my existing remote port 9997 telnet connection will get disconnected and I can no longer reconnect. Looking at the splunkd log, I can see the following: WARN TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 300 seconds. INFO TcpInputProc - Stopping IPv4 port 9997 WARN TcpInputProc - Stopping all listening ports. Queues blocked for more than 300 seconds WARN TcpOutputFd - Connect to 172.16.1.1:9997 failed. No connection could be made because the target machine actively refused it. ERROR TcpOutputFd - Connection to host=172.16.1.1:9997 failed INFO TcpOutputProc - Detected connection to 172.16.1.1:9997 closed INFO TcpOutputProc - Will close stream to current indexer 172.16.1.1:9997 INFO TcpOutputProc - Closing stream for idx=172.16.1.1:9997 ... View more

I disabled the firewall using netsh on both machines but this did not resolve the problem. Interestingly, from the client, I can telnet to port 8000 of splunk-server. The connectivity problem is specific to port 9777. ... View more

Splunk-Server = indexer with "send to indexer" app + deployment server Splunk-Client = Windows host with universal forwarder Splunk-server has an outputs.conf based on the steps of 'Create the "send to indexer" app' section: http://docs.splunk.com/Documentation/MSApp/1.1.0/MSInfra/Createthesendtoindexerapp Netstat shows that port 9997 in splunk-server is being used by the local Splunkd Service. This instance is using ports 49178, 8191, 49305, 49160, & 9997. Another splunk instance is using ports 8191. I cannot find any suspicious entries on the outputs.conf files on splunk-client and the inputs.conf file on splunk-server. (I prefer not to post the whole content of the text file for now so not to spam the forum but can post specific sections upon request). No non-default settings have been introduced into the environment. I can telnet to splunk-server:9997 from the local machine but cannot from splunk-client with a "Could not open connection to the host, on port 9997: Connect failed" error. ... View more

Windows Firewall on the server is completed turned off. Doing a netstat -a on the server gives the following: TCP 172.16.1.1:9997 splunk-server:49556 ESTABLISHED TCP 172.16.1.1:9997 SPLUNK-CLIENT:49168 ESTABLISHED ... View more

TcpOutputFd - Connect to 172.16.1.1:9997 failed. No connection could be made because the target machine actively refused it. ERROR TcpOutputFd - Connection to host=172.16.1.1:9997 failed ... View more

Here is the indexer inputs.conf content: [default] host = splunk-server [splunktcp://9997] connection_host = none Here is the SplunkUniversalForwarder\etc\apps\sendtoindexer\local outputs.conf content: [tcpout] defaultGroup = default-autolb-group [tcpout:default-autolb-group] server = 172.16.1.1:9997 [tcpout-server://172.16.1.1:9997] From the splunkd.log file, I keep seeing the following: ERROR TcpOutputFd - Connection to host=172.16.1.1:9997 failed WARN TcpOutputProc - Applying quarantine to ip=172.16.1.1 port=9997 _numberOfFailures=2 ... View more