Getting Data In

heavy forwarder network traffic requirements

awurster
Contributor

hey guys -

a few questions we've got regarding the forwarding of indexed data between a heavy forwarder and indexers. we are deploying splunk across several regions which are connected across WAN links, and therefore are trying to be mindful in how we deploy the forwarders and what do expect.

  1. is there a formula for calculating bandwidth requirements for forwarded indexes? (i.e. the ratio of total index size to transfer size if using encryption and/or compression)
  2. when forwarding indexed data between indexers – is the transfer steady or bursted (scheduled and/or regular intervals)? (using this for WAN link capacity planning)
  3. is there some rule of thumb or calculation for calculating search traffic bandwidth utilisation to search peers? (would like to use this for determining whether to place indexers at remote branch sites or colocate them at the head office site)

adamw
Communicator

A heavy forwarder is aware of props, and therefore does compression of data before sending it. It's really hard to determine your bandwidth requirements, since it is completely dependent on your environment, what you're logging, is it raw log files, or scripted inputs, how many forwarders, what type of OS, chattiness of logs on that forwarder, etc...

That being said, Splunk is quite tolerant of network slowdowns/outages, especially when using multiple indexers and doing auto loadbalancing, which is highly recommended when you need to scale out indexers, or to ensure that your data is always being received somewhere.

awurster
Contributor

thanks adam. to clarify:

1 - i'm mostly curious as to what compression is used, to estimate the actual transfer sizes (regardless of speed).
3 - i'd like to see some sizing guidance regarding search traffic. for instance - what does an average splunk user generate per day in terms of search data - including queries and responses (there must be one because there is a one CPU per user guideline already).

as for #2 - this may already be answered... according to splunk, this is pretty much a FIFO queue so that as fast or steady as the data comes in, so will the data be put out.

Get Updates on the Splunk Community!

Thanks for the Memories! Splunk University, .conf24, and Community Connections

Thank you to everyone in the Splunk Community who joined us for .conf24 – starting with Splunk University and ...

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...