All Apps and Add-ons

Splunk App for Stream: Is there documentation on tuning for receiving 2gbps UDP packets without dropping packets?

kwchang_splunk
Splunk Employee
Splunk Employee

Hello.

I have a customer who wants to do a performance test using stream. He has 2 scenarios: 

First one is about collecting UDP packets without drops. UDP packets will be collected from 2ea x 1g NICs with 2gbps in total. The fields that need to be extracted are L4 attributes including src_ip, src_port, dest_ip, dest_port and timestamp. The measurement would be the number of droped packets so we need to minimize the drops. 

The other scenario is collecting TCPpackets. TCP packets will be generated with the speed of 35,000 connections/seconds. Also, he will check the number of the lost packet. 

Is there any guide documentation for tuning parameters of OS kernel and streamfwd for doing these kind of tests?

Thank you in advance.
  

0 Karma
1 Solution

vshcherbakov_sp
Splunk Employee
Splunk Employee

Hello,

Stream documentation has recommendations for setting the linux kernel parameters - http://docs.splunk.com/Documentation/StreamApp/6.3.2/DeployStreamApp/Deploymentrequirements#Linux

As for configuring the Stream Forwarder, I'd recommend increasing the number of processing threads to ~3-7 by setting the ProcessingThreads parameter in streamfwd.xml file (see docs for more details - http://docs.splunk.com/Documentation/StreamApp/6.3.2/DeployStreamApp/ConfigureStreamForwarder)

Other streamfwd.xml settings that you may need to configure:

PcapBufferSize: set to 67108864 (64MB) or more

MaxTcpSessionCount: may need to increase if the generated traffic load has more than 50000 concurrent sessions

TcpConnectionTimeout: set to a lower value to ~30 (seconds)

View solution in original post

vshcherbakov_sp
Splunk Employee
Splunk Employee

I just noticed that you're planning to capture 2Gbps on 2x1Gbps NICs. One thing I'd watch carefully in this case is SPAN/TAP/switch drop rate to make sure you're not overloading the NIC bandwidth with the test traffic.

Having poor quality data feed (missing packets) may cause extra Stream processing overhead (excessive memory and CPU usage) as well as poor event data quality (garbage in/garbage out). TCP traffic is more sensitive to the quality of the data feed, but it applies to UDP as well.

0 Karma

vshcherbakov_sp
Splunk Employee
Splunk Employee

Hello,

Stream documentation has recommendations for setting the linux kernel parameters - http://docs.splunk.com/Documentation/StreamApp/6.3.2/DeployStreamApp/Deploymentrequirements#Linux

As for configuring the Stream Forwarder, I'd recommend increasing the number of processing threads to ~3-7 by setting the ProcessingThreads parameter in streamfwd.xml file (see docs for more details - http://docs.splunk.com/Documentation/StreamApp/6.3.2/DeployStreamApp/ConfigureStreamForwarder)

Other streamfwd.xml settings that you may need to configure:

PcapBufferSize: set to 67108864 (64MB) or more

MaxTcpSessionCount: may need to increase if the generated traffic load has more than 50000 concurrent sessions

TcpConnectionTimeout: set to a lower value to ~30 (seconds)

vshcherbakov_sp
Splunk Employee
Splunk Employee

We haven't had a need to tune the net.ipv4.udp_mem parameter. However, there's obviously enough variance in OS versions/traffic load profile/etc. to potentially make your customer's test setup different enough from our internal to warrant different recommendations.

I'd recommend running the test with the settings specified in Stream's doc first to see if there's any need for further kernel settings tuning.

0 Karma

kwchang_splunk
Splunk Employee
Splunk Employee

Thank you vshcherbakov.

What do you think about "net.ipv4.udp_mem" for UDP?
Any recommendation?

0 Karma
*NEW* Splunk Love Promo!
Snag a $25 Visa Gift Card for Giving Your Review!

It's another Splunk Love Special! For a limited time, you can review one of our select Splunk products through Gartner Peer Insights and receive a $25 Visa gift card!

Review:





Or Learn More in Our Blog >>