I am getting following error message on universal forwarder logs:
11-10-2013 17:43:38.750 +0530 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=notconnected
11-10-2013 17:43:46.141 +0530 ERROR HTTPClient - Should have gotten at least 3 tokens in status line, while getting response code. Only got 0.
11-10-2013 17:43:46.141 +0530 INFO HttpPubSubConnection - Secure HTTP POST failed: Unknown read error
11-10-2013 17:43:46.141 +0530 INFO HttpPubSubConnection - Could not obtain connection, will retry after=83 seconds.
11-10-2013 17:43:50.750 +0530 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=notconnected
I'm getting the error too.
But only for deployment clients in a particular network zone.
I can telnet to the deployment server on TCP 8089 fine, but the clients get the errors above.
At this stage I think it is a routing issue, our firewall team has been involved but have not detected any drops.
Check also make sure the local firewall / iptables is permitting TCP8089 to the DS host, and since these are different zones also confirm that the actual clients can connect to 8089 and not just your machine.
After that, make sure there is twoway (inbound / outbound) traffic through the firewall / acl for 8089 to the DS enabled.
We had exactly the same issue with the same error message and we struggled to figure it out - this turns out to be a MTU setting issue with a data center switch. Makes sense, given the ability to telnet to a port, but the web service then fails to work.
This answer is unlikely to help in most cases, however, I was getting this error on my local laptop (lab) where I had Splunk Enterprise (Deployment server) and Splunk Universal Forwarder (UF) running with the UF's targetUri setting in deploymentclient.conf pointing to localhost (local machine's IP actually). The issue of course was they were both using 8089 for mgmt port. By changing the port on my Enterprise instance to 8091 and restarting the enterprise instance running the deployment server, issue was resolved. Use
./splunk set splunkd-port 8091 on my DS
Restart DS instance