Getting Data In

udp data packets lost at Heavy Forwarder

splunk4nisha
New Member

I am observing packet loss on Heavy forwarder due to which I am missing the important messages which we are being sent using snmp traps. I have already increased the rmem buffer size to the suggested value for splunk stream app on Splunk docs(which I thought should be more than enough) , but even after that change there are still a lot of packet drops on the HF.

current stats:

sysctl net.core.rmem_max
net.core.rmem_max = 33554432

netstats:
netstat -suna

Udp:
52071486 packets received
21017 packets to unknown port received.
3747277 packet receive errors
82100 packets sent
3747277 receive buffer errors
0 send buffer errors
UdpLite:
IpExt:
InNoRoutes: 27
InMcastPkts: 8
InOctets: 31643507863
OutOctets: 6061193400
InMcastOctets: 288
InNoECTPkts: 62078913
InECT0Pkts: 1301

Any idea, what should be the ideal size for the net.core.rmem_max that can guarantee receive buffer errors reduce to zero.
Or this is something which we cannot achieve by increase the buffer size?

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Based on your HF hardware capacity, set one of the below for the UDP input that you've:

queueSize = <integer>[KB|MB|GB]
* Maximum size of the in-memory input queue.
* Default: 500KB.

persistentQueueSize = <integer>[KB|MB|GB|TB]
* Maximum size of the persistent queue file.
* Persistent queues can help prevent loss of transient data. For information on
  persistent queues and how the 'queueSize' and 'persistentQueueSize' settings
  interact, search the online documentation for "persistent queues"..
* If you set this to a value other than 0, then 'persistentQueueSize' must
  be larger than either the in-memory queue size (as defined by the 'queueSize'
  setting in inputs.conf or 'maxSize' settings in [queue] stanzas in
  server.conf).
* Default: 0 (no persistent queue).
0 Karma

gcusello
SplunkTrust
SplunkTrust

In addition I suggest to use two Heavy forwarders with a Load balancer to distribute load and be sure of HA features!
Bye.
Giuseppe

0 Karma

wgawhh5hbnht
Communicator
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...