Problem with Hydra scheduler: sprayReadyJobs exception

[halr9000]

I am troubleshooting my VMware app configuration, and am faced with this error in splunkd.log:

014-04-08 19:24:52,596 ERROR [ta_vmware_collection_scheduler://puff] Problem with hydra scheduler ta_vmware_collection_scheduler://puff:
 Something went unstable with a Hydra asset, typically due to a REST timeout or misconfiguration, rebuilding and validating entities
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/SA-Hydra/bin/hydra/hydra_scheduler.py", line 1718, in run
    collection_manifest.sprayReadyJobs(self.node_manifest)
  File "/opt/splunk/etc/apps/SA-Hydra/bin/hydra/hydra_scheduler.py", line 512, in sprayReadyJobs
    raise ForceHydraRebuild
ForceHydraRebuild: Something went unstable with a Hydra asset, typically due to a REST timeout or misconfiguration, rebuilding and validating entities

Data does not appear to be coming in from the data collection node (DCN)

[halr9000 · Accepted solution]

Received an answer from the lab:

This is hydra's internal rebuild. It is run whenever it is incapable of collecting data. This usually means you have one DCN and thatn DCN became unresponsive. Scheduler had no one to give jobs to and rebuilt itself.

Essentially if we can't really do anything we just turn ourselves off and on again.

So it appears that the error indicates that Hydra is restarting itself because the DCN is inaccessible. Root cause could be network or system related on the DCN.