Deployment Architecture

DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected"

Vinesh93
Explorer

We have a client that connected to Splunk Deployment server where we get an error "DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected". Ping and telnet to the Deployment server works fine and when we try to add some data into splunk its not getting indexed due to the above error. Is there any possible solutions available for this scenario?

Tags (1)
0 Karma

Gavrikova
Engager

I read a lot of comments about the error DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected

To summarize what I found out:

  • This error is about the connection between DC and DS, so it doesn't affect the possibility of your forwarder to upload the logs. But if there's a problem with DC - DS communication your DC can be unable to deploy apps to your forwarders (like outputs or inputs), so your forwarder doesn't know where to take logs and where to forward.

What you can check on forwarder:

  1. Is your DS uri set properly: $SPLUNK_HOME/bin/splunk show deploy-poll
  2. Is there a network connection between DC and DS. For example using telnet <DS-uri> 8089 (if you didn't change management port)
  3. Is your forwarder showing on DS in clients tab and when it phoned home for the last time.
  4. Did your forwarder get the apps from DS. They shoud be located in directories like this: $SPLUNK_HOME/etc/apps/<your_app_name>/local/
  5. The version of Splunk forwarder.

In my case my forwarder wasn't showing on DS in clients tab, but the DS uri was correct. And there were no problems with the network connectivity. But I resolved the issue by upgrading the forwarder from 6.2.5 to 7.0.2. And on indexers I have 7.2.1.

Hope it's helpful.

0 Karma

tsaikumar009
Explorer

super in that case, the issue would be that in the monitor stanza for certain log paths might be different in that hosts. I faced the same issue, i cleared duplicate monitor stanzas for same log paths and logs started rolling post that.

0 Karma

tsaikumar009
Explorer

the connection from your server to DS should be two way on port 8089 only the apps will be deployed to the instances.

check the connection from Server to DS on 8089 and also from DS to Server on same port 8089. If both works this error should not occur.

0 Karma

Vinesh93
Explorer

@tsaikumar009 telnet from both end is fine but still im getting this error and new events are not getting indexed. Am I missing anything here ?

0 Karma

lakshman239
Influencer

I assume you are telnet'ing from both DS to your client and vice versa on the port 8089 (or as per your deployment). Did you restart the client? do you see any other errors/warning in client's splunkd.log?

0 Karma

Vinesh93
Explorer

Yes, telnet from both end is fine. Also restarted the client, even after restarting the client the same error "err=not_connected" is generated in splunkd.log and new events are not getting indexed.

0 Karma

mannyk1splunk
Loves-to-Learn

Hi All,

Have you got a resolution to this issue. We are facing similar issue.

 

0 Karma

vasunder
Loves-to-Learn Lots

Hi All,

Did anyone got a resolution to this issue. We are facing similar issue.

Thank You

0 Karma

codewire
Loves-to-Learn

I updated firewall rules to allow traffic traffic from DC and issue was resolved.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...