All Apps and Add-ons

Problem with Hydra scheduler: sprayReadyJobs exception

halr9000
Motivator

I am troubleshooting my VMware app configuration, and am faced with this error in splunkd.log:

014-04-08 19:24:52,596 ERROR [ta_vmware_collection_scheduler://puff] Problem with hydra scheduler ta_vmware_collection_scheduler://puff:
 Something went unstable with a Hydra asset, typically due to a REST timeout or misconfiguration, rebuilding and validating entities
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/SA-Hydra/bin/hydra/hydra_scheduler.py", line 1718, in run
    collection_manifest.sprayReadyJobs(self.node_manifest)
  File "/opt/splunk/etc/apps/SA-Hydra/bin/hydra/hydra_scheduler.py", line 512, in sprayReadyJobs
    raise ForceHydraRebuild
ForceHydraRebuild: Something went unstable with a Hydra asset, typically due to a REST timeout or misconfiguration, rebuilding and validating entities

Data does not appear to be coming in from the data collection node (DCN)

0 Karma
1 Solution

halr9000
Motivator

Received an answer from the lab:

This is hydra's internal rebuild. It is run whenever it is incapable of collecting data. This usually means you have one DCN and thatn DCN became unresponsive. Scheduler had no one to give jobs to and rebuilt itself.

Essentially if we can't really do anything we just turn ourselves off and on again.

So it appears that the error indicates that Hydra is restarting itself because the DCN is inaccessible. Root cause could be network or system related on the DCN.

View solution in original post

halr9000
Motivator

Received an answer from the lab:

This is hydra's internal rebuild. It is run whenever it is incapable of collecting data. This usually means you have one DCN and thatn DCN became unresponsive. Scheduler had no one to give jobs to and rebuilt itself.

Essentially if we can't really do anything we just turn ourselves off and on again.

So it appears that the error indicates that Hydra is restarting itself because the DCN is inaccessible. Root cause could be network or system related on the DCN.

Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...