I'm currently reviewing the Splunk deployment server as a possibility to manage 4 search heads and 10 indexers, and conceivably thousands of forwarders in the future. Since the potential is there, I'm considering this a large deployment and plan on using a dedicated instance of Splunk to manage. What is the minimum hardware i should be considering for this size deployment? Could I get by with using a VM?
Yes, a VM is fine. It doesn't need lots of resources since files only get deployed and nothing else is being done on it. Maybe in future with thousands of indexers some HW should be considered, but I guess you gonna see how it'll perform than.
I agree a VM should be fine. If you do end up having resource constraints, you can also tune how often the clients ping the deployment server for updates. By default it checks for updates every 30 seconds, but you could tune this to be 5/30/whatever mins for the forwarders, and have a much lower load on the deployment server. See the docs at: http://www.splunk.com/base/Documentation/latest/admin/ConfigureDeploymentClients, in particular the phoneHomeIntervalInSecs setting.
Simple recommendations for deployment server :
If you can, Use a dedicated deployment server.
or if you really have no choice, on linux you can have a second instance of splunk.
since Splunk 5.* this was greatly improved
A single DS can now handle more than 1000+ to 10000 clients, and the phonehome interval strategy is better.
however this is still a single threaded process.
A single deployment server can have trouble to server more than 500 clients.
It's recommended to have several deployment servers (on different instances or different boxes).
So what's the best solution for many forwarders? "Stacked" Deployment Servers? Or can you put a load balancer in front of several other deployment servers? Will the checksums match if clients hit different deployment servers each time?