Since Splunk Enterprise 9.2.0, Splunk has introduced "Deployment Server Scaling", which involves setting Deployment Servers behind a load balancer (or use DNS mapping) and granting all access to a single network share. Each DS uses the share path to update and share app configurations and post log files. This allows the DS' to keep apps, client lists and client status in sync between them. While Splunk documentation mentions 50 clients, this is only in reference to ensuring the DS is on its own server, not sharing functionality with any other Splunk instance such as search head, indexer, Monitoring Console, License Manger, etc. A Deployment Server can actually handle up to 25,000 clients, if granted enough system and network resources to manage the load. With Deployment Server scaling, the number of forwarders that can be managed multiplies with each Deployment Server added to the "cluster". Two can manage up to 50,000 clients, three can manage up to 75,000, etc. All Deployment Servers in a cluster share all apps and all clients. DS Scaling is also referred to as "clustering", though it works nothing like indexer or search head clusters-- the different DS's don't communicate with one another directly or formally form a "cluster". This allows very large environments to manage a multitude of forwarders. Too many forwarders? Add another Deployment Server. Here are a few links: Splunk Documentation: Implement a Deployment Server Cluster Splunk Documentation: Estimate Deployment Server Performance Deployment Server section of this Splunk Lantern article: Scaling your Splunk Deployment, which consolidates relevant Splunk documentation Splunk Community Article: Deployment Server Scalability Best Practices "Discovered Intelligence" blog article on setting up a Splunk Deployment Server cluster. I have not (yet) tested their suggestions but this is a great place to start for a quick overview of what's needed. Deployment Servers are on track for significant improvements in the near future as well, with the goal of reducing/eliminating the need for 3rd party tools such as Puppet or Ansible for those who wish to manage everything within Splunk itself.
... View more