I have one deployment server to service 4 HFs and 1 deployer to service 3 SHs in cluster.
What is the best way to push/deploy configurations/apps to HFs and SHs?
Also, for some reason, I don't see outputs.conf from any HFs. Is that managed by the DS as well?
Search Heads cannot necer be managed by the Deployment Server, only by Deployer.
DS is used to manage Heavy Forwarders and Universal Forwarders ( https://docs.splunk.com/Documentation/Splunk/8.0.0/Updating/Aboutdeploymentserver ).
Best practices hint to create a Technical Add-On (TA) that contains output.conf and deploymentclients.conf files, so when you install a Universal Forwarder, copy this TA in $SPLUNKHOME/etc/apps and restart Splunk, in this way the UF is managed by the DS, don't use the CLI commands or put outputs.conf and deploymentclients.conf in $SPLUNKHOME/etc/system/local.
At this point, that you have your UF correctly connected to the Deploymet Server and Indexers, you can create and deploy all the TAs you need to manage your inputs.
As I said Search Heads cannot never be managed by the DS, always and only with the Deployer ( https://docs.splunk.com/Documentation/Splunk/8.0.0/DistSearch/AboutSHC )!
So you copy the apps to deploy to SHs in the Deployer and then you push them to SHs ( https://docs.splunk.com/Documentation/Splunk/8.0.0/DistSearch/PropagateSHCconfigurationchanges ).
In Search Heads outputs.con it's usually in $SPLUNK_HOME/etc/system/local, anyway you don't need to manage it by the Deployer, you can do this by UI before to create the cluster.
It's also possible to manage Deployer's Apps with the DS but I don't hint to do: it's easier to manually manage it.
Thank you for sharing. The issue I am facing is to migrating an imperfect system previously designed and adding more servers to the picture. I am still loosing the guts 1x1.