This is a serious "it depends" question. A forwarder's bandwidth requirements depends greatly on the event volume going through it.
A Splunk Light Forwarder has a built-in governor on its bandwidth usage. The default for a single light forwarder is 256 kilobytes per second (set in $SPLUNK_HOME/etc/apps/SplunkLightForwarder/default/limits.conf)
Based on that alone, one could argue that each forwarder needs as much as 2 Megabits/sec of bandwidth - assuming the event volume is high enough to consistently use that much bandwidth. But if your event volume is sufficiently low, you don't need nearly that much...
The questions you should be asking yourself, based on the data you have is how many forwarders, producing how many messages per second each, with an average message size of how many bytes? Then you can begin to estimate your actual bandwidth needs.
If you set it low and the queue fills up on the forwarder, will it back off until it's sent the queue? Could you loose data?
when the forwarder pauses :
It will be able to resume where it stopped for all the persistent inputs
- log file monitoring (As long as the files do not rotate out of reach from splunk.)
- wineventlogs (using a counter)
- etc ...
- another forwarder (it will pause too in chain)
but the forwarder will not be able to resume from non persistent inputs, once the queue buffer is full, the new events will be dropped.
- tcp/udp inputs (like syslog)
- scripted inputs
the workaround are to use persistent queues or have your logs written to file, then monitor them.
You risk data "lost in flight" if 1) your network isn't suitable (10gb is preferred)
2) your maxkbps is too low
3) parsing queue is too low
There may not be a one size fits all answer. If you deploy a light weight forwarder it limits itself to 256 kilobytes / second by default. This is tunable in the limits.conf file
[thruput] maxKBps = 256
If the forwarder is configured too low it may fall behind a bit during times of elevated event generations.
Yes, i would like to know this as well:
If you set it low and the queue fills up on the forwarder, will it back off until it's sent the queue? Could you loose data? – Marinus Apr 30 at 8:31
Yes, there is a crompressed parameter in the outputs.conf file for non-ssl connections, or useClientSSLCompression for the SSL ones.