Cooked connection denotes communications between two Splunk nodes as opposed to Raw connections which refer to non-Splunk nodes passing their data to Splunk.
First we need to ensure that the forwarder is listening on the correct port. To determine this we simply issue a netstat:
netstat –an | grep 9997
If the forwarder is listening properly you should see a result similar to:
Tcp 0 0 0.0.0.0:9997 0.0.0.0:* LIST
This tells us that the forwarder is in listen mode for port 9998.
Now that we’ve determined we’re listening on the correct port we need to test the communications path between the forwarder and indexer. To do this we attempt to open a telnet session to the indexer from the forwarder:
telnet indexername.domainname.com 9997
If the port is available the connection should be successful almost immediately. Should the connection fail we’ll want to try another port to determine if we have a port availability issue or something more. So next we’ll attempt to telnet to port 8089 (Splunk management port which should always be open on an indexer):
telnet indexername.domainname.com 8089
If the connection is successful you should see something close to the following:
Connected to indexername.domainname.com
Once we’ve made the successful connection, we now know that there is an open port issue with port 9998 to the indexer. Now the question remains, is this an issue at the firewall layer or within a local firewall such as iptables on the indexer itself. To determine this, we SSH into the indexer and attempt to telnet back onto ourselves via port 9998.
telnet localhost 9997
If the problem lies with a local firewall such as iptables, you will receive an error similar to the following:
telnet: connect to address ::1: Connection refused
So what does this tell us? The refusal locally tells us that port 9998 has not been opened via the local firewall and this is the source of our issue. To resolve we simply open port 9998 via iptables (or your local firewall) to resolve.
iptables –D INPUT –p tcp –dport 9997 –j DROP
service iptables save
If the telnet connection to localhost connects successfully, you have determined that the communications issue is tied to the fact that port 9998 needs to be opened at the network firewall layer to allow communications with the forwarder.
... View more