We are running a SHC with Splunk Enterprise OnPrem 9.0.1 and noticed that the concurrent searches in one of the nodes is way higher than the rest (3 times aprox.) even though the scheduler delegation shows its delegating evenly across the nodes.
Most of the scheduled searches are from an app that runs dbx queries to keep updated some lookups, these are scheduled to run a few times a week but appear to be running constantly in the scheduler.
These concurrent searches run constantly even after a restart of the node.
It doesnt happen in a single instance with the same apps, so we think it is a clustering issue.
How can we troubleshoot/debug this behaviour?