Getting Data In
Highlighted

syslog udp drops in linux

New Member

Hi,

Architecture:

  1. We have syslog-ng running in our infra.
  2. This syslog resides behind a LB
  3. This alerts come to LB and from LB it gets processed at syslog-ng
  4. The syslog logs are forwarded to splunk

Issue:

  1. When the syslog-ng are sent to LB alerts are getting dropped, this was confirmed by doing a netstat -us

netstat -us
IcmpMsg:
InType3: 86
InType8: 63
InType11: 24
InType13: 6
InType17: 18
OutType0: 63
OutType3: 1719
OutType11: 3394
OutType14: 6
Udp:
3541784219 packets received
6194 packets to unknown port received.
13334858510 packet receive errors
3292282273 packets sent
13334858510 receive buffer errors
44 send buffer errors
UdpLite:
IpExt:
InNoRoutes: 3728
InMcastPkts: 6
InOctets: 13341061420695
OutOctets: 6925411558186
InMcastOctets: 216
InNoECTPkts: 20437111460
InECT0Pkts: 48

  1. The configuration in syslog-ng.conf is

cat syslog-ng.conf | more
@version: 3.5

options {
threaded (yes);
flushlines (0);
keep
timestamp (no);
stats-freq (600);
mark-freq (0);
timereopen (10);
log
fifosize (120000);
create
dirs (no);
keephostname (yes);
dir
perm(0755);
perm(0644);
chainhostnames(no);
normalize
hostnames(yes);
usedns (yes);
use
fqdn (yes);
dns_cache(yes);
};

source stcp { tcp(ip(10.xxx.xx.xx) port(6514) keep-alive(yes) tcp-keep-alive(yes) maxconnections(3000));
tcp(ip(10.xxx.xx.xx) port(6514) keep-alive(yes) tcp-keep-alive(yes) maxconnections(3000));
tcp(ip(10.xxx.xx.xx) port(6514) keep-alive(yes) tcp-keep-alive(yes) max
connections(3000));
tcp(ip(10.xxx.xx.xx) port(6514) keep-alive(yes) tcp-keep-alive(yes) maxconnections(3000));
tcp(ip(10.xxx.xx.xx) port(6514) keep-alive(yes) tcp-keep-alive(yes) max
connections(3000));
};

source sudp { udp(ip(10.xxx.xx.xx) port(6514) sorcvbuf(128000000));
udp(ip(10.xxx.xx.xx) port(6514));
udp(ip(10.xxx.xx.xx) port(6514));
udp(ip(10.xxx.xx.xx) port(6514));
udp(ip(10.xxx.xx.xx) port(6514));
};

source s_internal {
internal();
};

  1. cat /etc/sysctl.conf

To increase TCP max buffer size up to 128MB

net.core.rmemmax = 128000000
net.core.wmem
max = 128000000

  1. free -m
    total used free shared buff/cache available
    Mem: 64249 2914 530 4 60805 60650
    Swap: 3967 1097 2870

  2. Environment
    syslog-ng and snmp (looperng)
    RHEL 7.3

grep -c ^processor /proc/cpuinfo
16

64GB Memory

2TB disk

How do we reduce the "packet receive errors". Is there any tweaking that splunk can recommend so that we don't loose syslog data?

Thanks for any inputs

Dinesh

Tags (2)
0 Karma
Highlighted

Re: syslog udp drops in linux

New Member

Hi, here you can find some tips: https://support.oneidentity.com/technical-documents/syslog-ng-premium-edition/7.0.12/collecting-log-...

But unless there is a special reason why you must use UDP, I'd suggest using TCP instead.

0 Karma
Highlighted

Re: syslog udp drops in linux

New Member

We have almost like 7000 devices and its not possible to change to UDP.

Also I changed the net.core.rmem_max in the /etc/sysctl.conf

and also changed the local.conf entry for the so_rcvbuf value.

I still see drops. Is there any procedure I need to follow in order to come at the correct value that I should give in the rmem and so_rcvbuf?

0 Karma
Highlighted

Re: syslog udp drops in linux

Explorer

Hi,
- You can try increasing the UDP buffers (both kernel and syslog-ng side) even to multiple GBs.
- If that does not help, you can split the UDP traffic to even more different udp sources (different port is enough). Each source will have its own udp buffer so the chance to fill up decreases.
- The next thing to consider is to place new syslog-ng machines behind the LB.
- Last but not least I would suggest to place multiple syslog-ng relay servers as close as possible to the original UDP log sources (eg. one per each geolocation/subnet/etc) and send the logs to the central syslog-ngs through tcp.

0 Karma