Getting Data In

How much data should be sent to one forwarder?

nkingsbury
Engager

Hello,
I am setting up a log collector with a Universal Forwarder attached for collecting network logs (syslog-ng) and then sending them to Splunk Cloud.

I am wondering if there is a good rule of thumb/best practice as to how many devices, or how much data should be sent to one collector/forwarder.

I plan to collect logs from: 6 firewalls, 32 routers, 165 switches, as well as some software logs like Cisco ISE.

All of those devices are spread around the world. Should I set up collectors in regional data-centers, or would I be OK sending everything to one?

0 Karma
1 Solution

woodcock
Esteemed Legend

I would do 2 behind a load balancer to give you some fault-tolerance through redundancy. That load can be handled by just one when one of them dies.

View solution in original post

0 Karma

woodcock
Esteemed Legend

I would do 2 behind a load balancer to give you some fault-tolerance through redundancy. That load can be handled by just one when one of them dies.

0 Karma

nkingsbury
Engager

Thank you for the answer. It seems to be the common consensus that I should have a load balancer in front of my collectors. Let me spell this out a bit becuase I am quite new to this and I cant find documentation for exactly what I am doing.

So, I will point my network device syslogs at a load balancer, that load balancer is setup to send the traffic to two different syslog-ng/UF's, which then forward the logs up to the cloud indexers.

Is there a recommended load balancer product to use for this case?

0 Karma

MuS
Legend

Well, don't forget that your load balancer must be HA as well - and yes F5 does a pretty decent job in handling the Splunk traffic.

cheers, MuS

0 Karma

nkingsbury
Engager

are we talking a physical appliance? I was hopping something like HAProxy or LVS would suffice.

0 Karma

woodcock
Esteemed Legend

Yes, that will work, too.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

A better measure than number of devices is data rate. A UF should have no problem with 256 KB/s or more. IF you're still concerned, stand up multiple syslog-ng servers (with UFs) behind a load balancer.

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Get More Out of Your Security Practice With a SIEM

Get More Out of Your Security Practice With a SIEMWednesday, July 31, 2024  |  11AM PT / 2PM ETREGISTER ...