We have a client that connected to Splunk Deployment server where we get an error "DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected". Ping and telnet to the Deployment server works fine and when we try to add some data into splunk its not getting indexed due to the above error. Is there any possible solutions available for this scenario?
I assume you are telnet'ing from both DS to your client and vice versa on the port 8089 (or as per your deployment). Did you restart the client? do you see any other errors/warning in client's splunkd.log?
Yes, telnet from both end is fine. Also restarted the client, even after restarting the client the same error "err=not_connected" is generated in splunkd.log and new events are not getting indexed.
the connection from your server to DS should be two way on port 8089 only the apps will be deployed to the instances.
check the connection from Server to DS on 8089 and also from DS to Server on same port 8089. If both works this error should not occur.
@tsaikumar009 telnet from both end is fine but still im getting this error and new events are not getting indexed. Am I missing anything here ?
super in that case, the issue would be that in the monitor stanza for certain log paths might be different in that hosts. I faced the same issue, i cleared duplicate monitor stanzas for same log paths and logs started rolling post that.
I read a lot of comments about the error
DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
To summarize what I found out:
inputs), so your forwarder doesn't know where to take logs and where to forward.
What you can check on forwarder:
$SPLUNK_HOME/bin/splunk show deploy-poll
telnet <DS-uri> 8089(if you didn't change management port)
In my case my forwarder wasn't showing on DS in clients tab, but the DS uri was correct. And there were no problems with the network connectivity. But I resolved the issue by upgrading the forwarder from 6.2.5 to 7.0.2. And on indexers I have 7.2.1.
Hope it's helpful.