I am getting following error message on universal forwarder logs:
11-10-2013 17:43:38.750 +0530 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
11-10-2013 17:43:46.141 +0530 ERROR HTTPClient - Should have gotten at least 3 tokens in status line, while getting response code. Only got 0.
11-10-2013 17:43:46.141 +0530 INFO HttpPubSubConnection - Secure HTTP POST failed: Unknown read error
11-10-2013 17:43:46.141 +0530 INFO HttpPubSubConnection - Could not obtain connection, will retry after=83 seconds.
11-10-2013 17:43:50.750 +0530 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
That is really interesting since I am running the UF on the same host as the search head but I also have the management port for the UF turned off
disableDefaultPort = true
and it is talking to another server for its deployment information.
I am going to turn the UF off for a bit to see if it helps.
another common error message from the captain for each member that stops running scheduled searches
05-27-2017 15:02:32.502 -0500 ERROR SHCMasterArtifactHandler - failed on handle async replicate request sid=scheduler_adminxxxxRMD52a88c92ed83e8b0e_at_1495915200_11776_2E1C054F-9A8B-4D4A-BBC0-29F0562C7AED err='targetPeer="member", targetGuid="88275523-AE18-4CD9-AD67-7956E06449C1" cannot be valid target for artifactId=scheduleradmincaptain_RMD52a88c92ed83e8b0e_at_1495915200_11776_2E1C054F-9A8B-4D4A-BBC0-29F0562C7AED srcPeer="captain", srcGuid="A6A2F1D5-37C9-419C-A85E-A42376EDD483" reason="peer already has artifact"'
This answer is unlikely to help in most cases, however, I was getting this error on my local laptop (lab) where I had Splunk Enterprise (Deployment server) and Splunk Universal Forwarder (UF) running with the UF's targetUri setting in deploymentclient.conf pointing to localhost (local machine's IP actually). The issue of course was they were both using 8089 for mgmt port. By changing the port on my Enterprise instance to 8091 and restarting the enterprise instance running the deployment server, issue was resolved. Use
./splunk set splunkd-port 8091 on my DS
Restart DS instance
We had exactly the same issue with the same error message and we struggled to figure it out - this turns out to be a MTU setting issue with a data center switch. Makes sense, given the ability to telnet to a port, but the web service then fails to work.
Check also make sure the local firewall / iptables is permitting TCP8089 to the DS host, and since these are different zones also confirm that the actual clients can connect to 8089 and not just your machine.
After that, make sure there is twoway (inbound / outbound) traffic through the firewall / acl for 8089 to the DS enabled.
I'm getting the error too.
But only for deployment clients in a particular network zone.
I can telnet to the deployment server on TCP 8089 fine, but the clients get the errors above.
At this stage I think it is a routing issue, our firewall team has been involved but have not detected any drops.