Splunk App for VMWare: How does the Hydra scheduler cache requests?

[tnerpel_splunk]

All,

Can anyone give me insight into how the hydra scheduler used by the Splunk App for VMWare caches API requests during a vCenter server reboot? During a reboot the vCenter server is being slammed by requests from the DCN. Do we have any settings around controlling or modifying this?

In hydra_gateway.log and hydra_access.log I am seeing the following:

hydra_gateway.log
2017-01-19 09:03:14,624 INFO [HydraWSGI:8464] [CacheBatchProcessor] parsed cache batch of count=30
2017-01-19 09:03:14,630 INFO [HydraWSGI:8464] [HydraCacheManager] current cache_length=30, checking cache for outdated entries...

hydra_access.log
2017-01-19 09:03:14,633 INFO POST /hydra/cache/batch '200 OK' - - - 1ms
2017-01-19 09:04:55,558 INFO POST /hydra/cache/batch '200 OK' - - - 7ms

Thanks!

[tnerpel_splunk · Accepted solution]

Answer:

When the vCenter is rebooting the scheduler does not push jobs to the hydra queue in DCN because the authentication to vCenter fails. The scheduler will not be able to connect to the vCenter server while it is still fully coming back online and hence will not schedule jobs to the DCN until it is able to establish a connection and validate credentials of vCenter. The Scheduler should schedule jobs only when the server is in a state to accept API calls.