I am working on a Splunk Deployment with a cluster of search heads spanning two physical sites. At Site1 there is actually only one search head. At Site2 there are two search heads.
The load balancer managing Splunk Web access tends to favor Site1 and so almost all users end up landing on the sole search head at Site1 when they access our Splunk Web url.
What I have noticed, however, is that the search head at Site1 also seems to run almost all of the scheduled searches. Also, because it is favored by the load balancer for user access, users log into that search head and it takes on most of the ad-hoc searches.
I know that this setup is very non-optimal, and as the story goes, this is a mess I inherited recently in taking over Splunk responsibilities. More search heads are needed actually at both physical Sites, but in the meantime, I am trying to understand why the cluster captain is not more evenly distributing the saved searches, alerts, reports, etc. Why do they all seem to execute from the sole cluster member at Site1?
