UDP Drops on Linux

[sdwilkerson]

At a site, we are doing 5000-20000 EPS (Events Per Second) all udp/514 at each of 10 locations. At this time, we are required to do this all with a single server/indexer in each location.

This week we made syslog-ng as the listening service (instead of Splunk) since it seemed to work better regarding the other log recipients (see below). Syslog-ng listens on udp/514 and then:

• Forwards 100% via udp/514 to another direct-connected logging system
• Forwards a certain "match" of lt 1% to Netcool
• Writes 100% to a file

Splunk reads the file and seems to be performing well; however, the system reports many UDP errors indicating syslog-ng is not receiving it all. Tests with a traffic-generator (loggen bundled with syslog-ng) show that a fraction of what is sent gets processed by syslog-ng. A tcpdump at the same time shows the EXACT packet count that was sent was received, about 1/3 dropped by the kernel and about 1/3-1/2 of what was sent is what makes it the syslog-ng file.

Applying the tuning tweaks below have reduced the number of drops substantially; however, they have not been eliminated. Maybe it isn't possible in this situation, but I think it should be.

Questions

• Besides a packet capture, is there anything in RHEL (built-in preferred) that provides a counter of UDP packets attempted and not just those that are received (see comments on netstat -su output below)?
• Is it possible to receive all packets sent yet still get UDP errors?
• What is a reasonable amount of syslog EPS to expect with this hardware?
• Are there any other tweaks to the Kernel or Syslog-ng to make?

Environment

Indexer:

• RHEL 5.4
• 2 6core processors – 12 cores
• 24GB Memory
• 5TB SAS 15k disk

Data:

• Syslog (udp/514) from Palo Alto Firewall Data from several direct-connected firewalls (same switch)
• Messages are 250-350 bytes mostly, with some just over 400 and NONE over 500 bytes.

ERRORS

• netstat -su shows output like

Udp:
    3436531 packets received
    1553 packets to unknown port received.
    8402 packet receive errors
    109202 packets sent
    RcvbufErrors: 8402

• The "packets received" is how many were accepted by the listening application and NOT how many were attempted
• The "packets to unknown port" is what came in for the application when the application was down or not listening (e.g. during a restart)
• The "packet receive errors" clearly indicates an error; HOWEVER, it is not a one-for-one packets-to-error. You can have many more errors than packets were sent, indicating it is possible for a single packet to generate multiple errors.

Tuning Steps

• These are similar to what was done; however, multiple values were tried so the numbers below are not exactly what is in production now:
• We tried turning off the udp/514 forwarding to the other applications but we did not see a noticeable drop in errors
• kernel
• net.ipv4.udp_rmem_min = 131072
• net.ipv4.udp_wmem_min = 131072
• net.core.netdev_max_backlog=2000
• net.core.rmem_max=67108864
syslog-ng

options {
        sync (5000);
        time_reopen (10);
        time_reap(5);
        long_hostnames (off);
        use_dns (no);
        use_fqdn (no);
        create_dirs (no);
        keep_hostname (yes);
        log_fifo_size (536870912);
        stats_freq(60);
        flush_lines(500);
        flush_timeout(10000);
};