In our current enterprise topology, we are running Splunk 6.2.1 clustered search heads and we have our utility instances (cluster master, deployer, deployment server) running on the same host as one of our search heads.
Support has told us that we need to remove the deployer to a separate host so that it doesn't share with one of the search heads.
My question: is it sufficient to move the deployer to a separate host, or do other of the utility instances need to be moved?
Lastly, in addition to moving the deployer instance, we are looking to upgrade the search heads to v6.2.6.
Which should we perform first?
The docs specifically mention that you might be able to run all three of those roles on a single server, but it specifically says to not run the deployer on any of the search head instances.
Given that, you really need to move the deployer. If I were you, I'd build a new machine and migrate all the roles to it. Or create 3 somewhat smaller (but still within specs!) virtual machines and run each thing on its own server.
I would move instances around first, then upgrade them, though I don't think it matters much.
I agree with @rich7177's suggestion of using VMs for the utility stuff. That also allows you to take advantage of VMotion in a failure scenario.
The answer will always be "it depends" because it depends on the load the other utility services are handling. The deployment server likely be most chatty and is usually the first candidate to be moved to another host.
As long as the load/job of Deployment Server & cluster Master instance is less, its fine to have all in 3 in one. Offcourse having them in separate instances better incase if there's going to be more Forwarders or Indexers planned for near future which will increase the load on Deployment Server and Cluster Master.