udp data packets lost at Heavy Forwarder

[splunk4nisha]

I am observing packet loss on Heavy forwarder due to which I am missing the important messages which we are being sent using snmp traps. I have already increased the rmem buffer size to the suggested value for splunk stream app on Splunk docs(which I thought should be more than enough) , but even after that change there are still a lot of packet drops on the HF.

current stats:

sysctl net.core.rmem_max
net.core.rmem_max = 33554432

netstats:
netstat -suna

Udp:
52071486 packets received
21017 packets to unknown port received.
3747277 packet receive errors
82100 packets sent
3747277 receive buffer errors
0 send buffer errors
UdpLite:
IpExt:
InNoRoutes: 27
InMcastPkts: 8
InOctets: 31643507863
OutOctets: 6061193400
InMcastOctets: 288
InNoECTPkts: 62078913
InECT0Pkts: 1301

Any idea, what should be the ideal size for the net.core.rmem_max that can guarantee receive buffer errors reduce to zero.
Or this is something which we cannot achieve by increase the buffer size?

[somesoni2]

Based on your HF hardware capacity, set one of the below for the UDP input that you've:

queueSize = <integer>[KB|MB|GB]
* Maximum size of the in-memory input queue.
* Default: 500KB.

persistentQueueSize = <integer>[KB|MB|GB|TB]
* Maximum size of the persistent queue file.
* Persistent queues can help prevent loss of transient data. For information on
  persistent queues and how the 'queueSize' and 'persistentQueueSize' settings
  interact, search the online documentation for "persistent queues"..
* If you set this to a value other than 0, then 'persistentQueueSize' must
  be larger than either the in-memory queue size (as defined by the 'queueSize'
  setting in inputs.conf or 'maxSize' settings in [queue] stanzas in
  server.conf).
* Default: 0 (no persistent queue).

[gcusello]

In addition I suggest to use two Heavy forwarders with a Load balancer to distribute load and be sure of HA features!
Bye.
Giuseppe

[wgawhh5hbnht]

Have you tried enabling useACK=true
https://docs.splunk.com/Documentation/Splunk/latest/Admin/Outputsconf