Getting Data In

How much data should be sent to one forwarder?

nkingsbury
Engager

Hello,
I am setting up a log collector with a Universal Forwarder attached for collecting network logs (syslog-ng) and then sending them to Splunk Cloud.

I am wondering if there is a good rule of thumb/best practice as to how many devices, or how much data should be sent to one collector/forwarder.

I plan to collect logs from: 6 firewalls, 32 routers, 165 switches, as well as some software logs like Cisco ISE.

All of those devices are spread around the world. Should I set up collectors in regional data-centers, or would I be OK sending everything to one?

0 Karma
1 Solution

woodcock
Esteemed Legend

I would do 2 behind a load balancer to give you some fault-tolerance through redundancy. That load can be handled by just one when one of them dies.

View solution in original post

0 Karma

woodcock
Esteemed Legend

I would do 2 behind a load balancer to give you some fault-tolerance through redundancy. That load can be handled by just one when one of them dies.

View solution in original post

0 Karma

nkingsbury
Engager

Thank you for the answer. It seems to be the common consensus that I should have a load balancer in front of my collectors. Let me spell this out a bit becuase I am quite new to this and I cant find documentation for exactly what I am doing.

So, I will point my network device syslogs at a load balancer, that load balancer is setup to send the traffic to two different syslog-ng/UF's, which then forward the logs up to the cloud indexers.

Is there a recommended load balancer product to use for this case?

0 Karma

MuS
SplunkTrust
SplunkTrust

Well, don't forget that your load balancer must be HA as well - and yes F5 does a pretty decent job in handling the Splunk traffic.

cheers, MuS

0 Karma

nkingsbury
Engager

are we talking a physical appliance? I was hopping something like HAProxy or LVS would suffice.

0 Karma

woodcock
Esteemed Legend

Yes, that will work, too.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

A better measure than number of devices is data rate. A UF should have no problem with 256 KB/s or more. IF you're still concerned, stand up multiple syslog-ng servers (with UFs) behind a load balancer.

---
If this reply helps you, an upvote would be appreciated.
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!