i like the idea of search head pooling with respect to ease of managing configs across multiple search heads. but i'm a little confused as to how scheduled searches are divvied between the search heads in the pool. is there any logic behind it? will it work intelligently if i have non-homogeneous HW for my search heads?
Whichever search head happens to acquire a lock on a job instance first, they'll run it. So, no, there isn't a lot of intelligence behind it. If a machine is running slower (and assuming clocks are all in sync), then it will be slightly less likely to pick up a job, but that's about it.
We have 5 seachheads in our pool. 1 is dedicated to running scheduled searches. The others do NOT run scheduled searches. On the searchheads that you DON'T want to run scheduled searches, in .../splunk/etc/system/local in default-mode.conf add:
[pipeline:scheduler]
disabled = true
We had to do this locally on each searchhead because "disabled=false" doesn't work.
Basically you are setting up a single server as a Splunk jobs server.
Whichever search head happens to acquire a lock on a job instance first, they'll run it. So, no, there isn't a lot of intelligence behind it. If a machine is running slower (and assuming clocks are all in sync), then it will be slightly less likely to pick up a job, but that's about it.
are there any plans on the roadmap to set or influence which search head(s) in the search head pool run scheduled searches? I'd like to do this so i can have search head dedicated for ad-hoc searches and others for scheduled searches and still get the ease of maintenance you get with pooling search heads.