Getting Data In

How much data should be sent to one forwarder?

nkingsbury
Engager

Hello,
I am setting up a log collector with a Universal Forwarder attached for collecting network logs (syslog-ng) and then sending them to Splunk Cloud.

I am wondering if there is a good rule of thumb/best practice as to how many devices, or how much data should be sent to one collector/forwarder.

I plan to collect logs from: 6 firewalls, 32 routers, 165 switches, as well as some software logs like Cisco ISE.

All of those devices are spread around the world. Should I set up collectors in regional data-centers, or would I be OK sending everything to one?

0 Karma
1 Solution

woodcock
Esteemed Legend

I would do 2 behind a load balancer to give you some fault-tolerance through redundancy. That load can be handled by just one when one of them dies.

View solution in original post

0 Karma

woodcock
Esteemed Legend

I would do 2 behind a load balancer to give you some fault-tolerance through redundancy. That load can be handled by just one when one of them dies.

0 Karma

nkingsbury
Engager

Thank you for the answer. It seems to be the common consensus that I should have a load balancer in front of my collectors. Let me spell this out a bit becuase I am quite new to this and I cant find documentation for exactly what I am doing.

So, I will point my network device syslogs at a load balancer, that load balancer is setup to send the traffic to two different syslog-ng/UF's, which then forward the logs up to the cloud indexers.

Is there a recommended load balancer product to use for this case?

0 Karma

MuS
SplunkTrust
SplunkTrust

Well, don't forget that your load balancer must be HA as well - and yes F5 does a pretty decent job in handling the Splunk traffic.

cheers, MuS

0 Karma

nkingsbury
Engager

are we talking a physical appliance? I was hopping something like HAProxy or LVS would suffice.

0 Karma

woodcock
Esteemed Legend

Yes, that will work, too.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

A better measure than number of devices is data rate. A UF should have no problem with 256 KB/s or more. IF you're still concerned, stand up multiple syslog-ng servers (with UFs) behind a load balancer.

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...