Getting Data In
Highlighted

What is the minimum network bandwidth required for Splunk forwarding?

Splunk Employee
Splunk Employee

I plan to setup Splunk Forwarders to push Windows Events and also some linux events to my central Splunk indexer. Need to know what the minimum network bandwidth pipe is required for this?

Highlighted

Re: What is the minimum network bandwidth required for Splunk forwarding?

SplunkTrust
SplunkTrust

This is a serious "it depends" question. A forwarder's bandwidth requirements depends greatly on the event volume going through it.

A Splunk Light Forwarder has a built-in governor on its bandwidth usage. The default for a single light forwarder is 256 kilobytes per second (set in $SPLUNK_HOME/etc/apps/SplunkLightForwarder/default/limits.conf)

Based on that alone, one could argue that each forwarder needs as much as 2 Megabits/sec of bandwidth - assuming the event volume is high enough to consistently use that much bandwidth. But if your event volume is sufficiently low, you don't need nearly that much...

The questions you should be asking yourself, based on the data you have is how many forwarders, producing how many messages per second each, with an average message size of how many bytes? Then you can begin to estimate your actual bandwidth needs.

View solution in original post

Highlighted

Re: What is the minimum network bandwidth required for Splunk forwarding?

Communicator

If you set it low and the queue fills up on the forwarder, will it back off until it's sent the queue? Could you loose data?

0 Karma
Highlighted

Re: What is the minimum network bandwidth required for Splunk forwarding?

Splunk Employee
Splunk Employee

when the forwarder pauses :

It will be able to resume where it stopped for all the persistent inputs
- log file monitoring (As long as the files do not rotate out of reach from splunk.)
- wineventlogs (using a counter)
- etc ...
- another forwarder (it will pause too in chain)

but the forwarder will not be able to resume from non persistent inputs, once the queue buffer is full, the new events will be dropped.
- tcp/udp inputs (like syslog)
- scripted inputs
the workaround are to use persistent queues or have your logs written to file, then monitor them.

0 Karma
Highlighted

Re: What is the minimum network bandwidth required for Splunk forwarding?

Builder

You risk data "lost in flight" if 1) your network isn't suitable (10gb is preferred)
2) your maxkbps is too low
3) parsing queue is too low

0 Karma
Highlighted

Re: What is the minimum network bandwidth required for Splunk forwarding?

Splunk Employee
Splunk Employee

There may not be a one size fits all answer. If you deploy a light weight forwarder it limits itself to 256 kilobytes / second by default. This is tunable in the limits.conf file

[thruput]
maxKBps = 256

If the forwarder is configured too low it may fall behind a bit during times of elevated event generations.

View solution in original post

Highlighted

Re: What is the minimum network bandwidth required for Splunk forwarding?

Contributor

Yes, i would like to know this as well:

If you set it low and the queue fills up on the forwarder, will it back off until it's sent the queue? Could you loose data? – Marinus Apr 30 at 8:31

0 Karma
Highlighted

Re: What is the minimum network bandwidth required for Splunk forwarding?

Splunk Employee
Splunk Employee

could you post this as a separate question?

0 Karma
Highlighted

Re: What is the minimum network bandwidth required for Splunk forwarding?

New Member

Is there any compression used by the Forwarder on text based log files?

0 Karma
Highlighted

Re: What is the minimum network bandwidth required for Splunk forwarding?

Splunk Employee
Splunk Employee

Yes, there is a crompressed parameter in the outputs.conf file for non-ssl connections, or useClientSSLCompression for the SSL ones.

0 Karma