Deployment Architecture

How to use Heavy Forwarder servers as a Deployment Server?

jfeitosa_real
Path Finder

Is it possible to use Heavy Forwarder servers as a Deployment Server? Because we have a current implementation scenario that would be 15000 workstations sending logs through UF to 07 Heavy Forwarders.
I already have a Deployment Server, but I am afraid it will have CPU load problems and I wanted to use the 07 Heavy Forwarders as intermediary DS. It's possible?

Thanks in advance.

0 Karma
1 Solution

nickhills
Ultra Champion

It is possible, however.

Deployment servers have a very high connection count (one for each client/UF) and your HFs will also have high connection counts (at least one for each client/UF) - at your scale, each box would need to handle ~4300 connections

Splunk recommendation is that a single DS support max 500 clients, so you would be well advised to use 30 of them! (assuming 15k is not a typo!)

You can lessen the impact of DS client connections by reducing the DS client phone home interval (I have worked on environments which had 2500 clients per DS with a phone home interval of 10 mins) but at your scale, that is still 6.

In Short - given the large number of clients, and relatively small number of aggregating HFs I would not think co-locating the roles would be a sensible approach. I would use dedicated Deployment servers (and probably a few more HFs)

If my comment helps, please give it a thumbs up!

View solution in original post

manjunathmeti
SplunkTrust
SplunkTrust

Are you passing HEC tokens from deployment server to Heavy Forwarders?

0 Karma

jfeitosa_real
Path Finder

Yes, DS would manage Forwarders and Forwarders would deploy to UF on workstations.

0 Karma

nickhills
Ultra Champion

It is possible, however.

Deployment servers have a very high connection count (one for each client/UF) and your HFs will also have high connection counts (at least one for each client/UF) - at your scale, each box would need to handle ~4300 connections

Splunk recommendation is that a single DS support max 500 clients, so you would be well advised to use 30 of them! (assuming 15k is not a typo!)

You can lessen the impact of DS client connections by reducing the DS client phone home interval (I have worked on environments which had 2500 clients per DS with a phone home interval of 10 mins) but at your scale, that is still 6.

In Short - given the large number of clients, and relatively small number of aggregating HFs I would not think co-locating the roles would be a sensible approach. I would use dedicated Deployment servers (and probably a few more HFs)

If my comment helps, please give it a thumbs up!

jfeitosa_real
Path Finder

Hy nickhillscpl,

That's right, there are ~ 15K of workstations.
The project is not viable with this number of servers.
I did not find in the Splunk documentation, a topic talking about this limit of clients per DS.

Thanks for listening.

0 Karma

nickhills
Ultra Champion

It's a bit annoying that some of these values (for deployments at scale) are discussed only in Splunk training reference material, however there is some coverage of the topic here: https://wiki.splunk.com/Deploy:DeploymentServer
and here:
https://answers.splunk.com/answers/26620/how-many-clients-can-one-deployment-server-manage.html

If my comment helps, please give it a thumbs up!
0 Karma

jfeitosa_real
Path Finder

@nickhillscpl

I saw these links. There is really no documentation on this subject. Only in training is it debated.

Thanks for listening.

0 Karma

nickhills
Ultra Champion

And experience. I have worked on some large deployments, and it’s a real issue! As I say, you can run more clients per DS, but the trade off is the frequency you can have them check in. If you are seriously thinking of running 15k clients on a few DS servers you are going to have challenges.

If my comment helps, please give it a thumbs up!
0 Karma

jfeitosa_real
Path Finder

Certainly experience and acquired knowledge is everything.
I'm already giving up on this scenario. I will propose not to use deploy for workstations and only use for servers.

[]s
Thanks.

0 Karma

jfeitosa_real
Path Finder

Does the Heavy Forwarder have a Universal Forwarder client connection limit?
How many UFs can communicate with the HF?

Thanks,

0 Karma

nickhills
Ultra Champion

yes, but there is no pre-determined limit because it depends so much on the volume of data you are sending, and how much transforming and sorting that data needs on its way to indexing.

With roughly 2.5k UFs connected to each HF, you are going to need a decent amount of bandwidth on each HF to get events in from your UFs and out to your indexers.

I'd guess you could plan for that with 50 HFs and not be too overbuilt 🙂

If my comment helps, please give it a thumbs up!
0 Karma

jfeitosa_real
Path Finder

Oh my gosh! 50 HF? It is very very difficult! 😉

0 Karma

ChrisRyang
New Member

Just curious, how did your project go with DS/HF on the same server, thinking of doing the same, but the  deployment is much smaller. 

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...