Splunk App for VMWare - Hydra gateway not starting, no listening process on port 8008

[mikelanghorst]

A few weeks ago the OS was updated, and the Splunk HF was updated to 6.3. This was Splunk VMWare 3.1.0, but have updated to 3.2.1 to no affect.

I noticed today that neither the performance, nor inventory data is being collected. In the logs I see messages about not being able to connect:

2016-03-11 19:07:11,366 ERROR [ta_vmware_collection_worker://theta:10258] Problem with hydra worker ta_vmware_collection_worker://theta:10258: [HydraGatewayAdapter] could not authenticate with gateway after 3 retries
Traceback (most recent call last):
File "/app/splunk/etc/apps/SA-Hydra/bin/hydra/hydra_worker.py", line 507, in run
self.establishGateway()
File "/app/splunk/etc/apps/SA-Hydra/bin/hydra/hydra_worker.py", line 428, in establishGateway
hga = HydraGatewayAdapter(self.node_path, self.session_key, self.gateway_uri)
File "/app/splunk/etc/apps/SA-Hydra/bin/hydra/hydra_common.py", line 66, in init
self.authenticate_gateway()
File "/app/splunk/etc/apps/SA-Hydra/bin/hydra/hydra_common.py", line 83, in authenticate_gateway
raise Exception("[HydraGatewayAdapter] could not authenticate with gateway after %s retries" % (str(retry)))
Exception: [HydraGatewayAdapter] could not authenticate with gateway after 3 retries

Other indications that it's should be connecting on port 8008, but there's no listening process. troubleshooting ideas, or resolution?

[mikelanghorst · Accepted solution]

For 6.3, the issue ended up being for me the versions of the SA-Hyrda and SA-Utils had gotten out of sync, therefore couldn't communicate to be able to start the port 8008 service

[trapti_splunk]

Yes, we have an issue of SSL with Splunk 6.4 that does not connect to DCN on port 8008.

We already have a scheduled release of VMWare where we have fixed this.

[splunk24]

how to make 8008 port as SSL in vmware ? please help ..

[trapti_splunk]

Can you check below and update:

1) What is the Splunk version 6.3.x ?
2) Are you getting any error in logs specific to SSL or SSLv2?

[leesiangfong]

Hi Trapti,
My Splunk version is 6.4 and tested on 6.4 also has the same issue.

I do not get any error on SSL. May I know which log you referring to?

Thanks
Lee

[trapti_splunk]

Is your Splunk version different on SH/Indexer and DCN?

[mikelanghorst]

Same splunk version.

[leesiangfong]

My Splunk version is 6.4 (I also tested on 6.3) also face the same issue. Both DCN and SH/Indexer using the same version