Getting Data In

syslog udp drops in linux

dbashyam
Explorer

Hi,

Architecture:

  1. We have syslog-ng running in our infra.
  2. This syslog resides behind a LB
  3. This alerts come to LB and from LB it gets processed at syslog-ng
  4. The syslog logs are forwarded to splunk

Issue:

  1. When the syslog-ng are sent to LB alerts are getting dropped, this was confirmed by doing a netstat -us

netstat -us
IcmpMsg:
InType3: 86
InType8: 63
InType11: 24
InType13: 6
InType17: 18
OutType0: 63
OutType3: 1719
OutType11: 3394
OutType14: 6
Udp:
3541784219 packets received
6194 packets to unknown port received.
13334858510 packet receive errors
3292282273 packets sent
13334858510 receive buffer errors
44 send buffer errors
UdpLite:
IpExt:
InNoRoutes: 3728
InMcastPkts: 6
InOctets: 13341061420695
OutOctets: 6925411558186
InMcastOctets: 216
InNoECTPkts: 20437111460
InECT0Pkts: 48

  1. The configuration in syslog-ng.conf is

cat syslog-ng.conf | more
@version: 3.5

options {
threaded (yes);
flush_lines (0);
keep_timestamp (no);
stats-freq (600);
mark-freq (0);
time_reopen (10);
log_fifo_size (120000);
create_dirs (no);
keep_hostname (yes);
dir_perm(0755);
perm(0644);
chain_hostnames(no);
normalize_hostnames(yes);
use_dns (yes);
use_fqdn (yes);
dns_cache(yes);
};

source s_tcp { tcp(ip(10.xxx.xx.xx) port(6514) keep-alive(yes) tcp-keep-alive(yes) max_connections(3000));
tcp(ip(10.xxx.xx.xx) port(6514) keep-alive(yes) tcp-keep-alive(yes) max_connections(3000));
tcp(ip(10.xxx.xx.xx) port(6514) keep-alive(yes) tcp-keep-alive(yes) max_connections(3000));
tcp(ip(10.xxx.xx.xx) port(6514) keep-alive(yes) tcp-keep-alive(yes) max_connections(3000));
tcp(ip(10.xxx.xx.xx) port(6514) keep-alive(yes) tcp-keep-alive(yes) max_connections(3000));
};

source s_udp { udp(ip(10.xxx.xx.xx) port(6514) so_rcvbuf(128000000));
udp(ip(10.xxx.xx.xx) port(6514));
udp(ip(10.xxx.xx.xx) port(6514));
udp(ip(10.xxx.xx.xx) port(6514));
udp(ip(10.xxx.xx.xx) port(6514));
};

source s_internal {
internal();
};

  1. cat /etc/sysctl.conf

To increase TCP max buffer size up to 128MB

net.core.rmem_max = 128000000
net.core.wmem_max = 128000000

  1. free -m
    total used free shared buff/cache available
    Mem: 64249 2914 530 4 60805 60650
    Swap: 3967 1097 2870

  2. Environment
    syslog-ng and snmp (looperng)
    RHEL 7.3

grep -c ^processor /proc/cpuinfo
16

64GB Memory

2TB disk

How do we reduce the "packet receive errors". Is there any tweaking that splunk can recommend so that we don't loose syslog data?

Thanks for any inputs

Dinesh

Tags (2)
0 Karma

GergelyBodnar
Explorer

Hi,
- You can try increasing the UDP buffers (both kernel and syslog-ng side) even to multiple GBs.
- If that does not help, you can split the UDP traffic to even more different udp sources (different port is enough). Each source will have its own udp buffer so the chance to fill up decreases.
- The next thing to consider is to place new syslog-ng machines behind the LB.
- Last but not least I would suggest to place multiple syslog-ng relay servers as close as possible to the original UDP log sources (eg. one per each geolocation/subnet/etc) and send the logs to the central syslog-ngs through tcp.

0 Karma

frobert
New Member

Hi, here you can find some tips: https://support.oneidentity.com/technical-documents/syslog-ng-premium-edition/7.0.12/collecting-log-...

But unless there is a special reason why you must use UDP, I'd suggest using TCP instead.

0 Karma

dbashyam
Explorer

We have almost like 7000 devices and its not possible to change to UDP.

Also I changed the net.core.rmem_max in the /etc/sysctl.conf

and also changed the local.conf entry for the so_rcvbuf value.

I still see drops. Is there any procedure I need to follow in order to come at the correct value that I should give in the rmem and so_rcvbuf?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...