All Apps and Add-ons

Problem with Hydra scheduler: sprayReadyJobs exception

halr9000
Motivator

I am troubleshooting my VMware app configuration, and am faced with this error in splunkd.log:

014-04-08 19:24:52,596 ERROR [ta_vmware_collection_scheduler://puff] Problem with hydra scheduler ta_vmware_collection_scheduler://puff:
 Something went unstable with a Hydra asset, typically due to a REST timeout or misconfiguration, rebuilding and validating entities
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/SA-Hydra/bin/hydra/hydra_scheduler.py", line 1718, in run
    collection_manifest.sprayReadyJobs(self.node_manifest)
  File "/opt/splunk/etc/apps/SA-Hydra/bin/hydra/hydra_scheduler.py", line 512, in sprayReadyJobs
    raise ForceHydraRebuild
ForceHydraRebuild: Something went unstable with a Hydra asset, typically due to a REST timeout or misconfiguration, rebuilding and validating entities

Data does not appear to be coming in from the data collection node (DCN)

0 Karma
1 Solution

halr9000
Motivator

Received an answer from the lab:

This is hydra's internal rebuild. It is run whenever it is incapable of collecting data. This usually means you have one DCN and thatn DCN became unresponsive. Scheduler had no one to give jobs to and rebuilt itself.

Essentially if we can't really do anything we just turn ourselves off and on again.

So it appears that the error indicates that Hydra is restarting itself because the DCN is inaccessible. Root cause could be network or system related on the DCN.

View solution in original post

halr9000
Motivator

Received an answer from the lab:

This is hydra's internal rebuild. It is run whenever it is incapable of collecting data. This usually means you have one DCN and thatn DCN became unresponsive. Scheduler had no one to give jobs to and rebuilt itself.

Essentially if we can't really do anything we just turn ourselves off and on again.

So it appears that the error indicates that Hydra is restarting itself because the DCN is inaccessible. Root cause could be network or system related on the DCN.

Get Updates on the Splunk Community!

Synthetic Monitoring: Not your Grandma’s Polyester! Tech Talk: DevOps Edition

Register today and join TekStream on Tuesday, February 28 at 11am PT/2pm ET for a demonstration of Splunk ...

Instrumenting Java Websocket Messaging

Instrumenting Java Websocket MessagingThis article is a code-based discussion of passing OpenTelemetry trace ...

Announcing General Availability of Splunk Incident Intelligence!

Digital transformation is real! Across industries, companies big and small are going through rapid digital ...