Getting Data In

How much data should be sent to one forwarder?

nkingsbury
Engager

Hello,
I am setting up a log collector with a Universal Forwarder attached for collecting network logs (syslog-ng) and then sending them to Splunk Cloud.

I am wondering if there is a good rule of thumb/best practice as to how many devices, or how much data should be sent to one collector/forwarder.

I plan to collect logs from: 6 firewalls, 32 routers, 165 switches, as well as some software logs like Cisco ISE.

All of those devices are spread around the world. Should I set up collectors in regional data-centers, or would I be OK sending everything to one?

0 Karma
1 Solution

woodcock
Esteemed Legend

I would do 2 behind a load balancer to give you some fault-tolerance through redundancy. That load can be handled by just one when one of them dies.

View solution in original post

0 Karma

woodcock
Esteemed Legend

I would do 2 behind a load balancer to give you some fault-tolerance through redundancy. That load can be handled by just one when one of them dies.

0 Karma

nkingsbury
Engager

Thank you for the answer. It seems to be the common consensus that I should have a load balancer in front of my collectors. Let me spell this out a bit becuase I am quite new to this and I cant find documentation for exactly what I am doing.

So, I will point my network device syslogs at a load balancer, that load balancer is setup to send the traffic to two different syslog-ng/UF's, which then forward the logs up to the cloud indexers.

Is there a recommended load balancer product to use for this case?

0 Karma

MuS
Legend

Well, don't forget that your load balancer must be HA as well - and yes F5 does a pretty decent job in handling the Splunk traffic.

cheers, MuS

0 Karma

nkingsbury
Engager

are we talking a physical appliance? I was hopping something like HAProxy or LVS would suffice.

0 Karma

woodcock
Esteemed Legend

Yes, that will work, too.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

A better measure than number of devices is data rate. A UF should have no problem with 256 KB/s or more. IF you're still concerned, stand up multiple syslog-ng servers (with UFs) behind a load balancer.

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...