Recommended sizing for deployment server?

[rolltide]

We have a deployment server as an instance of a search head. How many clients can a deployment server can handle?

[trashyroadz]

Since Splunk Enterprise 9.2.0, Splunk has introduced "Deployment Server Scaling", which involves setting Deployment Servers behind a load balancer (or use DNS mapping) and granting all access to a single network share. Each DS uses the share path to update and share app configurations and post log files. This allows the DS' to keep apps, client lists and client status in sync between them.

While Splunk documentation mentions 50 clients, this is only in reference to ensuring the DS is on its own server, not sharing functionality with any other Splunk instance such as search head, indexer, Monitoring Console, License Manger, etc. A Deployment Server can actually handle up to 25,000 clients, if granted enough system and network resources to manage the load.

With Deployment Server scaling, the number of forwarders that can be managed multiplies with each Deployment Server added to the "cluster". Two can manage up to 50,000 clients, three can manage up to 75,000, etc. All Deployment Servers in a cluster share all apps and all clients.

DS Scaling is also referred to as "clustering", though it works nothing like indexer or search head clusters-- the different DS's don't communicate with one another directly or formally form a "cluster".  This allows very large environments to manage a multitude of forwarders. Too many forwarders? Add another Deployment Server.

Here are a few links:

• Splunk Documentation: Implement a Deployment Server Cluster
• Splunk Documentation: Estimate Deployment Server Performance
• Deployment Server section of this Splunk Lantern article: Scaling your Splunk Deployment, which consolidates relevant Splunk documentation
• Splunk Community Article: Deployment Server Scalability Best Practices
• "Discovered Intelligence" blog article on setting up a Splunk Deployment Server cluster. I have not (yet) tested their suggestions but this is a great place to start for a quick overview of what's needed.

Deployment Servers are on track for significant improvements in the near future as well, with the goal of reducing/eliminating the need for 3rd party tools such as Puppet or Ansible for those who wish to manage everything within Splunk itself.

[jimodonald]

The DS is a single-threaded application and does not scale well vertically. Adding more CPU/Memory to a DS is not going to help much. You'll need about 1 DS per (approximately) 5000 hosts. and you'll want to put them behind a load balancer.

the hardware requirements are pretty low. we have our sized at 8CPU/32GB of memory, but you can likely use smaller servers. I would recommend testing various sizing configs to see what works well for your environment.

[M_Radoszewski]

So for 100k forwarders that is 20 servers minimum, which doesn't sound even remotely maintainable assuming each of them will have a different set of configurations. Correct me if I am wrong, but this ultimately renders deployment servers useless and forces larger organisations to use different automated deployment tools, right?

[jimodonald]

with that many forwarders, you should be (IMHO) using GIT for managing your configs. and then you deploy the entire set of configs to all of the DS. think of managing them as a whole.

And with a deployment of that size, you should be talking to your Splunk Field Architect.

[ChrisG]

If you are deploying to more than 50 clients, your deployment server needs to be a separate Splunk Enterprise instance.

A dedicated deployment server can handle thousands of clients. There are numerous things to consider. See Estimate deployment server performance in Updating Splunk Enterprise Instances.

[M_Radoszewski]

The problem is - this is only suitable for small environments (up to 2000 forwarders). There are some environments out there that are 40+ times larger then covered in this document.

[trashyroadz]

A standalone Deployment Server that is not functioning with any other server roles should be able to handle up to 25,000 forwarders.

In the past, customers with large deployments have set up Deployment Servers behind a load balancer and kept apps sync'd between them using tools such as Puppet or Ansible.

Since 9.2.0, Splunk Deployment Servers have been architected to work as a "cluster" behind a load balancer and to keep apps and client status sync'd between them via a shared network directory. This allows any number of forwarders to be managed. For example, for an environment capturing data from 100,000 forwarders, a cluster of at least 4 Deployment Servers would be a good place to start.