Getting Data In

udp data packets lost at Heavy Forwarder

New Member

I am observing packet loss on Heavy forwarder due to which I am missing the important messages which we are being sent using snmp traps. I have already increased the rmem buffer size to the suggested value for splunk stream app on Splunk docs(which I thought should be more than enough) , but even after that change there are still a lot of packet drops on the HF.

current stats:

sysctl net.core.rmem_max
net.core.rmem_max = 33554432

netstat -suna

52071486 packets received
21017 packets to unknown port received.
3747277 packet receive errors
82100 packets sent
3747277 receive buffer errors
0 send buffer errors
InNoRoutes: 27
InMcastPkts: 8
InOctets: 31643507863
OutOctets: 6061193400
InMcastOctets: 288
InNoECTPkts: 62078913
InECT0Pkts: 1301

Any idea, what should be the ideal size for the net.core.rmem_max that can guarantee receive buffer errors reduce to zero.
Or this is something which we cannot achieve by increase the buffer size?

0 Karma

Revered Legend

Based on your HF hardware capacity, set one of the below for the UDP input that you've:

queueSize = <integer>[KB|MB|GB]
* Maximum size of the in-memory input queue.
* Default: 500KB.

persistentQueueSize = <integer>[KB|MB|GB|TB]
* Maximum size of the persistent queue file.
* Persistent queues can help prevent loss of transient data. For information on
  persistent queues and how the 'queueSize' and 'persistentQueueSize' settings
  interact, search the online documentation for "persistent queues"..
* If you set this to a value other than 0, then 'persistentQueueSize' must
  be larger than either the in-memory queue size (as defined by the 'queueSize'
  setting in inputs.conf or 'maxSize' settings in [queue] stanzas in
* Default: 0 (no persistent queue).
0 Karma


In addition I suggest to use two Heavy forwarders with a Load balancer to distribute load and be sure of HA features!

0 Karma

Path Finder
0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.