A few weeks ago the OS was updated, and the Splunk HF was updated to 6.3. This was Splunk VMWare 3.1.0, but have updated to 3.2.1 to no affect.
I noticed today that neither the performance, nor inventory data is being collected. In the logs I see messages about not being able to connect:
2016-03-11 19:07:11,366 ERROR [tavmwarecollectionworker://theta:10258] Problem with hydra worker tavmwarecollectionworker://theta:10258: [HydraGatewayAdapter] could not authenticate with gateway after 3 retries
Traceback (most recent call last):
File "/app/splunk/etc/apps/SA-Hydra/bin/hydra/hydraworker.py", line 507, in run
File "/app/splunk/etc/apps/SA-Hydra/bin/hydra/hydraworker.py", line 428, in establishGateway
hga = HydraGatewayAdapter(self.nodepath, self.sessionkey, self.gatewayuri)
File "/app/splunk/etc/apps/SA-Hydra/bin/hydra/hydracommon.py", line 66, in init
File "/app/splunk/etc/apps/SA-Hydra/bin/hydra/hydracommon.py", line 83, in authenticate_gateway
raise Exception("[HydraGatewayAdapter] could not authenticate with gateway after %s retries" % (str(retry)))
Exception: [HydraGatewayAdapter] could not authenticate with gateway after 3 retries
Other indications that it's should be connecting on port 8008, but there's no listening process. troubleshooting ideas, or resolution?
My Splunk version is 6.4 (I also tested on 6.3) also face the same issue. Both DCN and SH/Indexer using the same version
Can you check below and update:
1) What is the Splunk version 6.3.x ?
2) Are you getting any error in logs specific to SSL or SSLv2?
My Splunk version is 6.4 and tested on 6.4 also has the same issue.
I do not get any error on SSL. May I know which log you referring to?
Yes, we have an issue of SSL with Splunk 6.4 that does not connect to DCN on port 8008.
We already have a scheduled release of VMWare where we have fixed this.
For 6.3, the issue ended up being for me the versions of the SA-Hyrda and SA-Utils had gotten out of sync, therefore couldn't communicate to be able to start the port 8008 service