All Apps and Add-ons

Problem with Hydra scheduler: sprayReadyJobs exception

halr9000
Motivator

I am troubleshooting my VMware app configuration, and am faced with this error in splunkd.log:

014-04-08 19:24:52,596 ERROR [ta_vmware_collection_scheduler://puff] Problem with hydra scheduler ta_vmware_collection_scheduler://puff:
 Something went unstable with a Hydra asset, typically due to a REST timeout or misconfiguration, rebuilding and validating entities
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/SA-Hydra/bin/hydra/hydra_scheduler.py", line 1718, in run
    collection_manifest.sprayReadyJobs(self.node_manifest)
  File "/opt/splunk/etc/apps/SA-Hydra/bin/hydra/hydra_scheduler.py", line 512, in sprayReadyJobs
    raise ForceHydraRebuild
ForceHydraRebuild: Something went unstable with a Hydra asset, typically due to a REST timeout or misconfiguration, rebuilding and validating entities

Data does not appear to be coming in from the data collection node (DCN)

0 Karma
1 Solution

halr9000
Motivator

Received an answer from the lab:

This is hydra's internal rebuild. It is run whenever it is incapable of collecting data. This usually means you have one DCN and thatn DCN became unresponsive. Scheduler had no one to give jobs to and rebuilt itself.

Essentially if we can't really do anything we just turn ourselves off and on again.

So it appears that the error indicates that Hydra is restarting itself because the DCN is inaccessible. Root cause could be network or system related on the DCN.

View solution in original post

halr9000
Motivator

Received an answer from the lab:

This is hydra's internal rebuild. It is run whenever it is incapable of collecting data. This usually means you have one DCN and thatn DCN became unresponsive. Scheduler had no one to give jobs to and rebuilt itself.

Essentially if we can't really do anything we just turn ourselves off and on again.

So it appears that the error indicates that Hydra is restarting itself because the DCN is inaccessible. Root cause could be network or system related on the DCN.

Get Updates on the Splunk Community!

New Learning Videos on Topics Most Requested by You! Plus This Month’s New Splunk ...

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

How I Instrumented a Rust Application Without Knowing Rust

As a technical writer, I often have to edit or create code snippets for Splunk's distributions of ...

Splunk Community Platform Survey

Hey Splunk Community, Starting today, the community platform may prompt you to participate in a survey. The ...