WARN TcpOutputProc - Forwarding to indexer group proidx blocked for 249600 seconds. in splunkd.log?

[pavanae]

After configuring everything I couldn't able to index the data while was checking in the splunkd.log. I could see the following warnings occuring repeatedly

01-11-2016 14:46:25.760 -0500 WARN TcpOutputProc - Cooked connection to ip=10.200.32.13:9997 timed out
01-11-2016 14:46:38.951 -0500 WARN TcpOutputProc - Forwarding to indexer group proidx blocked for 249600 seconds.

What does that mean?How can i index the data sucessfully?

[jbsplunk · Accepted solution]

Cooked connection to ip=10.200.32.13:9997 timed out

Means that a SYN has been sent to establish a tcp connection, but no ACK was received in response to the SYN. This could be due to network trouble, a firewall, router, switch, or other general connectivity problem.

I would first check to ensure that 10.200.32.13 is actually listening on port 9997 via a tool like netstat.

If it is, I'd suggest manually connecting via telnet or netcat to the port from the forwarding box. Chances are this won't be successful, so then you'd need to examine the infrastructure via a network capture using a tool like tcpdump to validate that the data is sent, and has arrived at the destination.

Unfortunately, the error is fairly generic and as such only general advise can be provided.