DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected"

[Vinesh93]

We have a client that connected to Splunk Deployment server where we get an error "DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected". Ping and telnet to the Deployment server works fine and when we try to add some data into splunk its not getting indexed due to the above error. Is there any possible solutions available for this scenario?

[Gavrikova]

I read a lot of comments about the error DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected

To summarize what I found out:

• This error is about the connection between DC and DS, so it doesn't affect the possibility of your forwarder to upload the logs. But if there's a problem with DC - DS communication your DC can be unable to deploy apps to your forwarders (like outputs or inputs), so your forwarder doesn't know where to take logs and where to forward.

What you can check on forwarder:

1. Is your DS uri set properly: $SPLUNK_HOME/bin/splunk show deploy-poll
2. Is there a network connection between DC and DS. For example using telnet <DS-uri> 8089 (if you didn't change management port)
3. Is your forwarder showing on DS in clients tab and when it phoned home for the last time.
4. Did your forwarder get the apps from DS. They shoud be located in directories like this: $SPLUNK_HOME/etc/apps/<your_app_name>/local/
5. The version of Splunk forwarder.

In my case my forwarder wasn't showing on DS in clients tab, but the DS uri was correct. And there were no problems with the network connectivity. But I resolved the issue by upgrading the forwarder from 6.2.5 to 7.0.2. And on indexers I have 7.2.1.

Hope it's helpful.

[tsaikumar009]

super in that case, the issue would be that in the monitor stanza for certain log paths might be different in that hosts. I faced the same issue, i cleared duplicate monitor stanzas for same log paths and logs started rolling post that.

[tsaikumar009]

the connection from your server to DS should be two way on port 8089 only the apps will be deployed to the instances.

check the connection from Server to DS on 8089 and also from DS to Server on same port 8089. If both works this error should not occur.

[Vinesh93]

@tsaikumar009 telnet from both end is fine but still im getting this error and new events are not getting indexed. Am I missing anything here ?

[lakshman239]

I assume you are telnet'ing from both DS to your client and vice versa on the port 8089 (or as per your deployment). Did you restart the client? do you see any other errors/warning in client's splunkd.log?

[Vinesh93]

Yes, telnet from both end is fine. Also restarted the client, even after restarting the client the same error "err=not_connected" is generated in splunkd.log and new events are not getting indexed.

[mannyk1splunk]

Hi All,

Have you got a resolution to this issue. We are facing similar issue.

[vasunder]

Hi All,

Did anyone got a resolution to this issue. We are facing similar issue.

Thank You

[codewire]

I updated firewall rules to allow traffic traffic from DC and issue was resolved.