Deployment Architecture

How to use Heavy Forwarder servers as a Deployment Server?

Path Finder

Is it possible to use Heavy Forwarder servers as a Deployment Server? Because we have a current implementation scenario that would be 15000 workstations sending logs through UF to 07 Heavy Forwarders.
I already have a Deployment Server, but I am afraid it will have CPU load problems and I wanted to use the 07 Heavy Forwarders as intermediary DS. It's possible?

Thanks in advance.

0 Karma
1 Solution

Ultra Champion

It is possible, however.

Deployment servers have a very high connection count (one for each client/UF) and your HFs will also have high connection counts (at least one for each client/UF) - at your scale, each box would need to handle ~4300 connections

Splunk recommendation is that a single DS support max 500 clients, so you would be well advised to use 30 of them! (assuming 15k is not a typo!)

You can lessen the impact of DS client connections by reducing the DS client phone home interval (I have worked on environments which had 2500 clients per DS with a phone home interval of 10 mins) but at your scale, that is still 6.

In Short - given the large number of clients, and relatively small number of aggregating HFs I would not think co-locating the roles would be a sensible approach. I would use dedicated Deployment servers (and probably a few more HFs)

View solution in original post

Influencer

Are you passing HEC tokens from deployment server to Heavy Forwarders?

0 Karma

Path Finder

Yes, DS would manage Forwarders and Forwarders would deploy to UF on workstations.

0 Karma

Ultra Champion

It is possible, however.

Deployment servers have a very high connection count (one for each client/UF) and your HFs will also have high connection counts (at least one for each client/UF) - at your scale, each box would need to handle ~4300 connections

Splunk recommendation is that a single DS support max 500 clients, so you would be well advised to use 30 of them! (assuming 15k is not a typo!)

You can lessen the impact of DS client connections by reducing the DS client phone home interval (I have worked on environments which had 2500 clients per DS with a phone home interval of 10 mins) but at your scale, that is still 6.

In Short - given the large number of clients, and relatively small number of aggregating HFs I would not think co-locating the roles would be a sensible approach. I would use dedicated Deployment servers (and probably a few more HFs)

View solution in original post

Path Finder

Hy nickhillscpl,

That's right, there are ~ 15K of workstations.
The project is not viable with this number of servers.
I did not find in the Splunk documentation, a topic talking about this limit of clients per DS.

Thanks for listening.

0 Karma

Ultra Champion

It's a bit annoying that some of these values (for deployments at scale) are discussed only in Splunk training reference material, however there is some coverage of the topic here: https://wiki.splunk.com/Deploy:DeploymentServer
and here:
https://answers.splunk.com/answers/26620/how-many-clients-can-one-deployment-server-manage.html

0 Karma

Path Finder

@nickhillscpl

I saw these links. There is really no documentation on this subject. Only in training is it debated.

Thanks for listening.

0 Karma

Ultra Champion

And experience. I have worked on some large deployments, and it’s a real issue! As I say, you can run more clients per DS, but the trade off is the frequency you can have them check in. If you are seriously thinking of running 15k clients on a few DS servers you are going to have challenges.

0 Karma

Path Finder

Certainly experience and acquired knowledge is everything.
I'm already giving up on this scenario. I will propose not to use deploy for workstations and only use for servers.

[]s
Thanks.

0 Karma

Path Finder

Does the Heavy Forwarder have a Universal Forwarder client connection limit?
How many UFs can communicate with the HF?

Thanks,

0 Karma

Ultra Champion

yes, but there is no pre-determined limit because it depends so much on the volume of data you are sending, and how much transforming and sorting that data needs on its way to indexing.

With roughly 2.5k UFs connected to each HF, you are going to need a decent amount of bandwidth on each HF to get events in from your UFs and out to your indexers.

I'd guess you could plan for that with 50 HFs and not be too overbuilt 🙂

0 Karma

Path Finder

Oh my gosh! 50 HF? It is very very difficult! 😉

0 Karma