You could have a "main" DS that deploys only to subordinate DS nodes and have a cron job on each subordinate DS that runs every few minutes and synchronizes everything in
$SPLUNK_HOME/etc/deployment-apps and then hand-manage (or use load-balancer/VIP) to have your forwarders spread out across each of your subordinate DS instances.
So if I do that, forwarder management is going to be hit or miss right? Effectively every subordinate DS is a standalone? The "main" DS has the server classes, but I won't see what clients are connected? I'm guessing the information about clients doesn't live in /etc?
You should be able to get the client connection information from
_internal on your Search Head.
The documentation talks about how DS performance can break down with relatively few clients/connections here:
Unfortunately it really doesn't tell how to mitigate other than to change