topic How to use Heavy Forwarder servers as a Deployment Server? in Deployment Architecture

How to use Heavy Forwarder servers as a Deployment Server?

jfeitosa_real — Tue, 18 Feb 2020 15:25:41 GMT

Is it possible to use Heavy Forwarder servers as a Deployment Server? Because we have a current implementation scenario that would be 15000 workstations sending logs through UF to 07 Heavy Forwarders.
I already have a Deployment Server, but I am afraid it will have CPU load problems and I wanted to use the 07 Heavy Forwarders as intermediary DS. It's possible?

Thanks in advance.

Re: How to use Heavy Forwarder servers as a Deployment Server?

nickhills — Tue, 18 Feb 2020 15:38:04 GMT

It is possible, however.

Deployment servers have a very high connection count (one for each client/UF) and your HFs will also have high connection counts (at least one for each client/UF) - at your scale, each box would need to handle ~4300 connections

Splunk recommendation is that a single DS support max 500 clients, so you would be well advised to use 30 of them! (assuming 15k is not a typo!)

You can lessen the impact of DS client connections by reducing the DS client phone home interval (I have worked on environments which had 2500 clients per DS with a phone home interval of 10 mins) but at your scale, that is still 6.

In Short - given the large number of clients, and relatively small number of aggregating HFs I would not think co-locating the roles would be a sensible approach. I would use dedicated Deployment servers (and probably a few more HFs)

Re: How to use Heavy Forwarder servers as a Deployment Server?

manjunathmeti — Tue, 18 Feb 2020 15:38:56 GMT

Are you passing HEC tokens from deployment server to Heavy Forwarders?

Re: How to use Heavy Forwarder servers as a Deployment Server?

jfeitosa_real — Tue, 18 Feb 2020 16:40:48 GMT

Hy nickhillscpl,

That's right, there are ~ 15K of workstations.
The project is not viable with this number of servers.
I did not find in the Splunk documentation, a topic talking about this limit of clients per DS.

Thanks for listening.

Re: How to use Heavy Forwarder servers as a Deployment Server?

jfeitosa_real — Tue, 18 Feb 2020 16:43:07 GMT

Yes, DS would manage Forwarders and Forwarders would deploy to UF on workstations.

Re: How to use Heavy Forwarder servers as a Deployment Server?

nickhills — Tue, 18 Feb 2020 16:44:47 GMT

It's a bit annoying that some of these values (for deployments at scale) are discussed only in Splunk training reference material, however there is some coverage of the topic here: https://wiki.splunk.com/Deploy:DeploymentServer
and here:
https://answers.splunk.com/answers/26620/how-many-clients-can-one-deployment-server-manage.html

Re: How to use Heavy Forwarder servers as a Deployment Server?

jfeitosa_real — Tue, 18 Feb 2020 19:27:08 GMT

@nickhillscpl

I saw these links. There is really no documentation on this subject. Only in training is it debated.

Thanks for listening.

Re: How to use Heavy Forwarder servers as a Deployment Server?

nickhills — Tue, 18 Feb 2020 19:42:53 GMT

And experience. I have worked on some large deployments, and it’s a real issue! As I say, you can run more clients per DS, but the trade off is the frequency you can have them check in. If you are seriously thinking of running 15k clients on a few DS servers you are going to have challenges.

Re: How to use Heavy Forwarder servers as a Deployment Server?

jfeitosa_real — Tue, 18 Feb 2020 19:58:04 GMT

Certainly experience and acquired knowledge is everything.
I'm already giving up on this scenario. I will propose not to use deploy for workstations and only use for servers.

[]s
Thanks.

Re: How to use Heavy Forwarder servers as a Deployment Server?

jfeitosa_real — Tue, 18 Feb 2020 20:16:37 GMT

Does the Heavy Forwarder have a Universal Forwarder client connection limit?
How many UFs can communicate with the HF?

Thanks,

Re: How to use Heavy Forwarder servers as a Deployment Server?

nickhills — Tue, 18 Feb 2020 20:52:27 GMT

yes, but there is no pre-determined limit because it depends so much on the volume of data you are sending, and how much transforming and sorting that data needs on its way to indexing.

With roughly 2.5k UFs connected to each HF, you are going to need a decent amount of bandwidth on each HF to get events in from your UFs and out to your indexers.

I'd guess you could plan for that with 50 HFs and not be too overbuilt 🙂

Re: How to use Heavy Forwarder servers as a Deployment Server?

jfeitosa_real — Tue, 18 Feb 2020 21:06:20 GMT

Oh my gosh! 50 HF? It is very very difficult! 😉

Re: How to use Heavy Forwarder servers as a Deployment Server?

ChrisRyang — Fri, 30 Apr 2021 16:53:35 GMT

Just curious, how did your project go with DS/HF on the same server, thinking of doing the same, but the deployment is much smaller.