Is it possible to use Heavy Forwarder servers as a Deployment Server? Because we have a current implementation scenario that would be 15000 workstations sending logs through UF to 07 Heavy Forwarders.
I already have a Deployment Server, but I am afraid it will have CPU load problems and I wanted to use the 07 Heavy Forwarders as intermediary DS. It's possible?
Thanks in advance.
It is possible, however.
Deployment servers have a very high connection count (one for each client/UF) and your HFs will also have high connection counts (at least one for each client/UF) - at your scale, each box would need to handle ~4300 connections
Splunk recommendation is that a single DS support max 500 clients, so you would be well advised to use 30 of them! (assuming 15k is not a typo!)
You can lessen the impact of DS client connections by reducing the DS client phone home interval (I have worked on environments which had 2500 clients per DS with a phone home interval of 10 mins) but at your scale, that is still 6.
In Short - given the large number of clients, and relatively small number of aggregating HFs I would not think co-locating the roles would be a sensible approach. I would use dedicated Deployment servers (and probably a few more HFs)
That's right, there are ~ 15K of workstations.
The project is not viable with this number of servers.
I did not find in the Splunk documentation, a topic talking about this limit of clients per DS.
Thanks for listening.
It's a bit annoying that some of these values (for deployments at scale) are discussed only in Splunk training reference material, however there is some coverage of the topic here: https://wiki.splunk.com/Deploy:DeploymentServer
I saw these links. There is really no documentation on this subject. Only in training is it debated.
Thanks for listening.
And experience. I have worked on some large deployments, and it’s a real issue! As I say, you can run more clients per DS, but the trade off is the frequency you can have them check in. If you are seriously thinking of running 15k clients on a few DS servers you are going to have challenges.
Certainly experience and acquired knowledge is everything.
I'm already giving up on this scenario. I will propose not to use deploy for workstations and only use for servers.
Does the Heavy Forwarder have a Universal Forwarder client connection limit?
How many UFs can communicate with the HF?
yes, but there is no pre-determined limit because it depends so much on the volume of data you are sending, and how much transforming and sorting that data needs on its way to indexing.
With roughly 2.5k UFs connected to each HF, you are going to need a decent amount of bandwidth on each HF to get events in from your UFs and out to your indexers.
I'd guess you could plan for that with 50 HFs and not be too overbuilt 🙂