Hardware requirements for deployment server in large deployment?

[carmackd]

I'm currently reviewing the Splunk deployment server as a possibility to manage 4 search heads and 10 indexers, and conceivably thousands of forwarders in the future. Since the potential is there, I'm considering this a large deployment and plan on using a dedicated instance of Splunk to manage. What is the minimum hardware i should be considering for this size deployment? Could I get by with using a VM?

[yannK]

A single deployment server can have trouble to server more than 500 clients.
It's recommended to have several deployment servers (on different instances or different boxes).

[vbumgarner]

So what's the best solution for many forwarders? "Stacked" Deployment Servers? Or can you put a load balancer in front of several other deployment servers? Will the checksums match if clients hit different deployment servers each time?

[yannK · Accepted solution]

Simple recommendations for deployment server :

If you can, Use a dedicated deployment server.
or if you really have no choice, on linux you can have a second instance of splunk.

• to run several splunk intance, change the httpport and mgmtHostPort in $SPLUNK_HOME/etc/system/local/web.conf http://www.splunk.com/base/Documentation/4.1.7/Admin/Webconf
  

License

• on splunk 4.1.- you can use the forwarder license for the dedicated deployment server
• on splunk 4.2 make it a license.slave

Server settings

• Configure the deployment server by editing $SPLUNK_HOME/etc/system/local/server-class.conf
• Tune the server ulimit to accommodate for the number of open sockets
  http://answers.splunk.com/questions/13313/how-to-tune-ulimit-on-my-server/13314#13314

Client Settings

• The deployment period can be increased if you have more and more clients. phoneHomeIntervalInSecs in deploymentclient.conf on the clients http://www.splunk.com/base/Documentation/latest/Admin/Deploymentclientconf

[yannK]

since Splunk 5.* this was greatly improved
A single DS can now handle more than 1000+ to 10000 clients, and the phonehome interval strategy is better.
however this is still a single threaded process.

[wdhathaway]

I agree a VM should be fine. If you do end up having resource constraints, you can also tune how often the clients ping the deployment server for updates. By default it checks for updates every 30 seconds, but you could tune this to be 5/30/whatever mins for the forwarders, and have a much lower load on the deployment server. See the docs at: http://www.splunk.com/base/Documentation/latest/admin/ConfigureDeploymentClients, in particular the phoneHomeIntervalInSecs setting.

[LCM]

Yes, a VM is fine. It doesn't need lots of resources since files only get deployed and nothing else is being done on it. Maybe in future with thousands of indexers some HW should be considered, but I guess you gonna see how it'll perform than.