Why am I getting "TcpOutputProc - Channel not registered yet. Connection not available" in the splunkd.log?

[bouchardk]

Hi,

I'm new to the world of splunk. I'm on the 6.1.3 version.
I configured my Indexer and my Forwarder according to the splunk documentation. I got some problems and I found my answers on this forum and on google.
But when I check the splunkd.log, I see that a channel has not been registered. I can't find what I forgot.
I don't have ERROR that my SSL has not been correctly configured so I think that it's ok for this.

Thank you very much for your help

On my Indexer, I enable my port, so I have this :

tcp        0      0 *:8090                      *:*                         LISTEN

I configure the splunk logs to DEBUG but when I disable the DEBUG mode for the logs, I got INFO "Cooked connection ... timed out"
Here is my splunkd.log :

01-06-2015 11:01:58.835 -0500 DEBUG TcpOutputProc - AutoLB timer started to select new connection
01-06-2015 11:01:58.835 -0500 DEBUG TcpOutputProc - BEGIN - randomizeConnectionsList
01-06-2015 11:01:58.835 -0500 DEBUG TcpOutputProc - Indexer uri [Indexer IP]:8090, client refCount=0, client=NULL
01-06-2015 11:01:58.836 -0500 DEBUG TcpOutputProc - BEGIN - After sorting
01-06-2015 11:01:58.836 -0500 DEBUG TcpOutputProc - Indexer uri [Indexer IP]:8090, client refCount=0, client=NULL
01-06-2015 11:01:58.836 -0500 DEBUG TcpOutputProc - Found a candidate indexer which is currently not connected. [Indexer IP]:8090, client refCount=0, client=NULL
01-06-2015 11:01:58.836 -0500 DEBUG TcpOutputProc - getting connected clients
01-06-2015 11:01:58.836 -0500 DEBUG TcpOutputProc - channel not registered yet
01-06-2015 11:01:58.836 -0500 DEBUG TcpOutputProc - Connection not available. Waiting for connection ...
01-06-2015 11:01:58.836 -0500 DEBUG TcpOutputProc - ---- existing clients - start ----
01-06-2015 11:01:58.836 -0500 DEBUG TcpOutputProc - ---- existing clients - end ----
01-06-2015 11:01:58.836 -0500 DEBUG TcpOutputProc - channel not registered yet
01-06-2015 11:01:58.836 -0500 DEBUG TcpOutputProc - Connection not available. Waiting for connection ...
01-06-2015 11:01:58.836 -0500 DEBUG TcpOutputProc - Connector::runCookedStateMachine in state=eInit for [Indexer IP]:8090
01-06-2015 11:01:58.836 -0500 DEBUG TcpOutputProc - tcpConnect to [Indexer IP]:8090
01-06-2015 11:01:59.837 -0500 DEBUG TcpOutputProc - channel not registered yet
01-06-2015 11:01:59.837 -0500 DEBUG TcpOutputProc - Connection not available. Waiting for connection ...

Forwarder outputs.conf :

[tcpout]
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = (_audit|_introspection)
forwardedindex.filter.disable = false
autoLB = true
maxQueueSize = auto
disabled = false
defaultGroup = mdm
server = Indexer:8090

[tcpout:mdm]
compressed = false

[tcpout-server://Indexer:8090]
sslCertPath = $SPLUNK_HOME/etc/auth/server.pem
sslPassword = $1$w2bPHFJpZqfE
sslRootCAPath = $SPLUNK_HOME/etc/auth/cacert.pem
sslVerifyServerCert = false

Indexer inputs.conf :

[splunktcp-ssl:8090]
compressed = false

[SSL]
password = $1$2+3yldmmdYWN
requireClientCert = false
rootCA = $SPLUNK_HOME/etc/auth/cacert.pem
serverCert = $SPLUNK_HOME/etc/auth/server.pem

[neelamssantosh]

1.Check if the communication/ping/Handshake is happening between both.
Telnet forwarder to indexer 8090

2 . Check ,port are open and firewall is not blocking them.See listening connection
netstat -tnap|grep 8090
3. Use ./splunk list monitor
4. See metric.log for errors in forwarders.
5. Splunkd.log for connection establishment

[bouchardk]

Thanks for your help.

I tried to do the step 1 and apparently, a firewall between the both server blocked my port.
After open the port on firewall, I've seen some pushed event on my splunkd.log

Am I supposed to see always a registred channel ? I got a "unregistred channel for", this is problematic for something ?

01-16-2015 10:23:27.815 -0500 DEBUG TcpOutputProc - channel not registered yet
01-16-2015 10:23:27.816 -0500 DEBUG TcpOutputProc - Registering Channel for : source::/opt/splunkfw/var/log/splunk/splunkd.log|host::Indexer|splunkd|45Indexer:8090, oneTimeClient=0, _events.size()=0, _refCount=2, _waitingAckQ.size()=0, _supportsACK=0, _lastHBRecvTime=Fri Jan 16 10:23:27 2015
01-16-2015 10:23:27.816 -0500 DEBUG TcpOutputProc - Pushed eventId=2105 on chanID=5 to back of tcp client (tcp output) queue
01-16-2015 10:23:27.816 -0500 DEBUG TcpOutputProc - channel registered
01-16-2015 10:23:27.816 -0500 DEBUG TcpOutputProc - Unregistering Channel for : source::/opt/splunkfw/var/log/splunk/splunkd.log|host::Indexer|splunkd|45Indexer:8090, oneTimeClient=0, _events.size()=0, _refCount=3, _waitingAckQ.size()=0, _supportsACK=0, _lastHBRecvTime=Fri Jan 16 10:23:27 2015

Another question about the Indexer that receive the logs from my forwarder, how and where I can see in the command line on the indexer server that my logs has been received completely ?