topic Re: syslog udp drops in linux in Getting Data In

syslog udp drops in linux

dbashyam — Tue, 29 Sep 2020 22:44:10 GMT

Hi,

Architecture:

We have syslog-ng running in our infra.
This syslog resides behind a LB
This alerts come to LB and from LB it gets processed at syslog-ng
The syslog logs are forwarded to splunk

Issue:

When the syslog-ng are sent to LB alerts are getting dropped, this was confirmed by doing a netstat -us

netstat -us
IcmpMsg:
InType3: 86
InType8: 63
InType11: 24
InType13: 6
InType17: 18
OutType0: 63
OutType3: 1719
OutType11: 3394
OutType14: 6
Udp:
3541784219 packets received
6194 packets to unknown port received.
13334858510 packet receive errors
3292282273 packets sent
13334858510 receive buffer errors
44 send buffer errors
UdpLite:
IpExt:
InNoRoutes: 3728
InMcastPkts: 6
InOctets: 13341061420695
OutOctets: 6925411558186
InMcastOctets: 216
InNoECTPkts: 20437111460
InECT0Pkts: 48

The configuration in syslog-ng.conf is

cat syslog-ng.conf | more
@version: 3.5

options {
threaded (yes);
flush_lines (0);
keep_timestamp (no);
stats-freq (600);
mark-freq (0);
time_reopen (10);
log_fifo_size (120000);
create_dirs (no);
keep_hostname (yes);
dir_perm(0755);
perm(0644);
chain_hostnames(no);
normalize_hostnames(yes);
use_dns (yes);
use_fqdn (yes);
dns_cache(yes);
};

source s_tcp { tcp(ip(10.xxx.xx.xx) port(6514) keep-alive(yes) tcp-keep-alive(yes) max_connections(3000));
tcp(ip(10.xxx.xx.xx) port(6514) keep-alive(yes) tcp-keep-alive(yes) max_connections(3000));
tcp(ip(10.xxx.xx.xx) port(6514) keep-alive(yes) tcp-keep-alive(yes) max_connections(3000));
tcp(ip(10.xxx.xx.xx) port(6514) keep-alive(yes) tcp-keep-alive(yes) max_connections(3000));
tcp(ip(10.xxx.xx.xx) port(6514) keep-alive(yes) tcp-keep-alive(yes) max_connections(3000));
};

source s_udp { udp(ip(10.xxx.xx.xx) port(6514) so_rcvbuf(128000000));
udp(ip(10.xxx.xx.xx) port(6514));
udp(ip(10.xxx.xx.xx) port(6514));
udp(ip(10.xxx.xx.xx) port(6514));
udp(ip(10.xxx.xx.xx) port(6514));
};

source s_internal {
internal();
};

cat /etc/sysctl.conf

To increase TCP max buffer size up to 128MB

net.core.rmem_max = 128000000
net.core.wmem_max = 128000000

free -m
total used free shared buff/cache available
Mem: 64249 2914 530 4 60805 60650
Swap: 3967 1097 2870
Environment
syslog-ng and snmp (looperng)
RHEL 7.3

grep -c ^processor /proc/cpuinfo
16

64GB Memory

2TB disk

How do we reduce the "packet receive errors". Is there any tweaking that splunk can recommend so that we don't loose syslog data?

Thanks for any inputs

Dinesh

Re: syslog udp drops in linux

frobert — Thu, 10 Jan 2019 09:19:41 GMT

Hi, here you can find some tips: https://support.oneidentity.com/technical-documents/syslog-ng-premium-edition/7.0.12/collecting-log-messages-from-udp-sources

But unless there is a special reason why you must use UDP, I'd suggest using TCP instead.

Re: syslog udp drops in linux

dbashyam — Tue, 22 Jan 2019 08:13:53 GMT

We have almost like 7000 devices and its not possible to change to UDP.

Also I changed the net.core.rmem_max in the /etc/sysctl.conf

and also changed the local.conf entry for the so_rcvbuf value.

I still see drops. Is there any procedure I need to follow in order to come at the correct value that I should give in the rmem and so_rcvbuf?

Re: syslog udp drops in linux

GergelyBodnar — Tue, 22 Jan 2019 09:40:44 GMT

Hi,
- You can try increasing the UDP buffers (both kernel and syslog-ng side) even to multiple GBs.
- If that does not help, you can split the UDP traffic to even more different udp sources (different port is enough). Each source will have its own udp buffer so the chance to fill up decreases.
- The next thing to consider is to place new syslog-ng machines behind the LB.
- Last but not least I would suggest to place multiple syslog-ng relay servers as close as possible to the original UDP log sources (eg. one per each geolocation/subnet/etc) and send the logs to the central syslog-ngs through tcp.