How to troubleshoot why I'm getting handshake errors in forwarder logs?

[sunnyparmar]

Hi,

I am trying to push data to Splunk server from my workstation, but getting the INFO & WARN logs below in the splunkd.log file.
I had telnet the splunk ip as well with port and it is able to telnet, so please suggest what to do in this case as I am not able to upload data on Splunk through my PC.

11-04-2015 12:46:49.925 +0530 INFO  TcpOutputProc - Connected to idx=10.0.28.208:9997
11-04-2015 12:46:55.959 +0530 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
11-04-2015 12:47:07.960 +0530 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
11-04-2015 12:47:19.962 +0530 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
11-04-2015 12:47:26.963 +0530 WARN  HttpPubSubConnection - Unable to parse message from PubSubSvr: 
11-04-2015 12:47:26.963 +0530 INFO  HttpPubSubConnection - Could not obtain connection, will retry after=38 seconds.
11-04-2015 12:47:31.963 +0530 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
11-04-2015 12:47:43.964 +0530 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
11-04-2015 12:47:55.965 +0530 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
11-04-2015 12:48:07.966 +0530 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected

Thanks

[lguinn2]

I assume you have installed a Splunk forwarder on your workstation, based on your logs.

It appears that your Splunk forwarder has a valid outputs.conf based on the TcpOutputProc message in the splunkd.log
You can validate this by running this search on the indexer: index=_internal TcpOutputProc over the last 15 minutes. Examine the events, particularly the host field; I expect that you will find events from the workstation. This proves that your Splunk forwarder is able to send data to the indexer; it sent its own internal logs.

The error messages in splunkd.log indicate that your workstation is not able to talk to the deployment server (aka forwarder management). Somewhere on your forwarder, you will find a file named deploymentclient.conf. This file probably contains a typographical error. An error in the port number is also common. Once you correct the problem, restart the forwarder.

Hopefully, fixing deploymentclient.conf will allow the forwarder to download the apps it needs from the deployment server. Assuming that these apps configure inputs on your forwarder, you should start to see "real" data on your indexer and not just the internal forwarder logs.