Hi,
Architecture:
Issue:
netstat -us
IcmpMsg:
InType3: 86
InType8: 63
InType11: 24
InType13: 6
InType17: 18
OutType0: 63
OutType3: 1719
OutType11: 3394
OutType14: 6
Udp:
3541784219 packets received
6194 packets to unknown port received.
13334858510 packet receive errors
3292282273 packets sent
13334858510 receive buffer errors
44 send buffer errors
UdpLite:
IpExt:
InNoRoutes: 3728
InMcastPkts: 6
InOctets: 13341061420695
OutOctets: 6925411558186
InMcastOctets: 216
InNoECTPkts: 20437111460
InECT0Pkts: 48
cat syslog-ng.conf | more
@version: 3.5
options {
threaded (yes);
flush_lines (0);
keep_timestamp (no);
stats-freq (600);
mark-freq (0);
time_reopen (10);
log_fifo_size (120000);
create_dirs (no);
keep_hostname (yes);
dir_perm(0755);
perm(0644);
chain_hostnames(no);
normalize_hostnames(yes);
use_dns (yes);
use_fqdn (yes);
dns_cache(yes);
};
source s_tcp { tcp(ip(10.xxx.xx.xx) port(6514) keep-alive(yes) tcp-keep-alive(yes) max_connections(3000));
tcp(ip(10.xxx.xx.xx) port(6514) keep-alive(yes) tcp-keep-alive(yes) max_connections(3000));
tcp(ip(10.xxx.xx.xx) port(6514) keep-alive(yes) tcp-keep-alive(yes) max_connections(3000));
tcp(ip(10.xxx.xx.xx) port(6514) keep-alive(yes) tcp-keep-alive(yes) max_connections(3000));
tcp(ip(10.xxx.xx.xx) port(6514) keep-alive(yes) tcp-keep-alive(yes) max_connections(3000));
};
source s_udp { udp(ip(10.xxx.xx.xx) port(6514) so_rcvbuf(128000000));
udp(ip(10.xxx.xx.xx) port(6514));
udp(ip(10.xxx.xx.xx) port(6514));
udp(ip(10.xxx.xx.xx) port(6514));
udp(ip(10.xxx.xx.xx) port(6514));
};
source s_internal {
internal();
};
To increase TCP max buffer size up to 128MB
net.core.rmem_max = 128000000
net.core.wmem_max = 128000000
free -m
total used free shared buff/cache available
Mem: 64249 2914 530 4 60805 60650
Swap: 3967 1097 2870
Environment
syslog-ng and snmp (looperng)
RHEL 7.3
grep -c ^processor /proc/cpuinfo
16
64GB Memory
2TB disk
How do we reduce the "packet receive errors". Is there any tweaking that splunk can recommend so that we don't loose syslog data?
Thanks for any inputs
Dinesh
Hi,
- You can try increasing the UDP buffers (both kernel and syslog-ng side) even to multiple GBs.
- If that does not help, you can split the UDP traffic to even more different udp sources (different port is enough). Each source will have its own udp buffer so the chance to fill up decreases.
- The next thing to consider is to place new syslog-ng machines behind the LB.
- Last but not least I would suggest to place multiple syslog-ng relay servers as close as possible to the original UDP log sources (eg. one per each geolocation/subnet/etc) and send the logs to the central syslog-ngs through tcp.
Hi, here you can find some tips: https://support.oneidentity.com/technical-documents/syslog-ng-premium-edition/7.0.12/collecting-log-...
But unless there is a special reason why you must use UDP, I'd suggest using TCP instead.
We have almost like 7000 devices and its not possible to change to UDP.
Also I changed the net.core.rmem_max in the /etc/sysctl.conf
and also changed the local.conf entry for the so_rcvbuf value.
I still see drops. Is there any procedure I need to follow in order to come at the correct value that I should give in the rmem and so_rcvbuf?