Best practices for Deployment Server sizing and maintenance for 10K clients?

[jtsplunk]

What are the hardware requirements for a Deployment Server with up to 10,000 clients? At what point should multiple Deployment Servers be used? Number of clients? Size of apps? Where should one look to diagnose issues? Certain log messages?

At what point does one determine a better option is Puppet or Chef?

There is some guidance of 50+ clients with this article, but this is a much larger deployment:

http://docs.splunk.com/Documentation/Splunk/6.2.4/Updating/Calculatedeploymentserverperformance

[woodcock]

Watch this video and pay special attention to "Hierarchical Deployments":

https://www.youtube.com/watch?v=3i3Sz3aPrts

[bmacias84]

This is a subjective question with a subjective answer. Personally I think you should use both the deployment server and chef.

I would use chef/ puppet to manage the installation and anything in system/local. Items that i thing should not be controlled through apps/ TAs.

For the deployment server I've managed 3k clients using a single deployment server, but with 15 min checkin intervals not the default 5min. The longer your checkin intervals the more server you should be able to support. Thought this make it difficult if you need to deploy rapid changes. Alternatively you can setup a tiered deployment server environment with a single master deployment server. This does add complexity.

The article you reference states 12 core 12GB can support up too 1000 clients with 5min checkin intervals could deploy updates in 9.5minutes with a 50MB payload. If you increase your checkin intervals 10k clients shouldn't be a problem.