Syslog-NG Configuration: Review & Advise

[0xAli]

Hi All,

I want to share with you the Syslog-NG configuration, you can review and advise.

[+] Syslog-ng Preparation
============================================
Check SELinux status
getenforce

sudo mkdir -p /var/log/syslog
sudo chown -R splunk:splunk /var/log/syslog
sudo chmod 750 /var/log/syslog

sudo tee /etc/sysctl.d/99-syslog-ng.conf > /dev/null << 'EOF'
# Syslog-ng UDP receive buffer tuning
net.core.rmem_max = 33554432
net.core.rmem_default = 16777216
net.core.netdev_max_backlog = 5000

# TCP keepalive - fast dead-peer detection after keepalived VIP failover
net.ipv4.tcp_keepalive_time = 120
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_keepalive_probes = 5
EOF


sudo sysctl --system
sudo sysctl net.core.rmem_max net.core.rmem_default net.core.netdev_max_backlog net.ipv4.tcp_keepalive_time

============================================
[+] Syslog-NG Configuration
===========================

sudo tee /etc/syslog-ng/conf.d/data_sources.conf > /dev/null << 'EOF'
###############################################################################
# GLOBAL OPTIONS
###############################################################################

options {
    log-fifo-size(200000);
    threaded(yes);
    use-dns(no);
    dns-cache(no);
	keep-hostname(no);
    chain-hostname(no);
    log-msg-size(65536);
};

###############################################################################
# RAW MESSAGE TEMPLATE
###############################################################################

template t_raw {
    template("$MSG\n");
    template_escape(no);
};

###############################################################################
# TCP SOURCES
###############################################################################
# FortiGate - TCP 1515
source s_fortigate {
    network(
        ip(0.0.0.0)
        transport(tcp)
        port(1515)
        flags(no-parse, flow-control)
        max-connections(100)
        log-iw-size(10000)
        so-keepalive(yes)
        so-rcvbuf(1048576)
    );
};

###############################################################################
# UDP SOURCES
###############################################################################
# F5 WAF - UDP 2514
source s_f5waf {
    network(
        ip(0.0.0.0)
        transport(udp)
        port(2514)
        flags(no-parse)
        so-rcvbuf(4194304)
    );
};

###############################################################################
# TCP DESTINATIONS
###############################################################################
# FortiGate
destination d_fortigate {
    file(
        "/var/log/syslog/fortigate/${SOURCEIP}/${SOURCEIP}-$YEAR-$MONTH-$DAY-$HOUR.log"
        create_dirs(yes)
        dir-owner("splunk")
        dir-group("splunk")
        dir-perm(0750)
        owner("splunk")
        group("splunk")
        perm(0640)
        template(t_raw)
        fsync(no)
    );
};

###############################################################################
# UDP DESTINATIONS
###############################################################################
# F5 WAF
destination d_f5waf {
    file(
        "/var/log/syslog/f5waf/${SOURCEIP}/${SOURCEIP}-$YEAR-$MONTH-$DAY-$HOUR.log"
        create_dirs(yes)
        dir-owner("splunk")
        dir-group("splunk")
        dir-perm(0750)
        owner("splunk")
        group("splunk")
        perm(0640)
        template(t_raw)
        fsync(no)
    );
};

###############################################################################
# LOG PATHS
###############################################################################
log {
    source(s_fortigate);
    destination(d_fortigate);
    flags(final);
};

log {
    source(s_f5waf);
    destination(d_f5waf);
    flags(final);
};
###############################################################################
# END
###############################################################################
EOF

============================================
[+] Validate & Start
============================================
sudo syslog-ng -s
sudo systemctl restart syslog-ng
sudo systemctl status syslog-ng --no-pager

sudo ss -lntup | grep 1514
ss -ulm sport = :2514
============================================
[+] Testing
============================================
loggen --size 500 --rate 1000 --interval 600 10.0.0.1 514

logger --server 127.0.0.1 --port 1514 --tcp "PALOALTO_TCP_TEST"

logger --server 127.0.0.1 --port 2514 --udp "PALOALTO_UDP_TEST"

echo '<134>date=2026-08-11 time=13:00:00 devname=PA01 type=TRAFFIC action=allow srcip=10.1.1.1 dstip=10.2.2.2' | nc -w1 10.0.4.160 1514
============================================

[masonreed11]

Looks good overall. Just fix the port mismatch: FortiGate is configured for TCP 1515, but your tests use 1514/514. Make the test ports match the configured sources, then verify with ss that 1515 and 2514 are listening.

[PickleRick]

Well, this is more of a syslog-ng question than a Splunk one so you might get more info about syslog-ng specific tuning and tweaking on some syslog-ng related mailing list/forum.

But.

I can drop in my three cents (because why not ;-))

1. I'm not (and I've never been) a big fan of the "separate port for everything" approach. Yes, I know that in the old times when we'd receive syslog directly on Splunk forwarder you'd have a separate port for each product so that you can assign different sourcetype to each port. With syslog-ng or rsyslog you can just use network-level metadata (source IP in this case) to assign target source/index/sourcetype to the event. I find this approach more tidy and the network teams like it because you only ever poke holes in firewalls for the typical 514 port instead of doing a wide range of ports.

2. Having said that - I'm not a big expert on syslog-ng and I'm not sure if you can in a reasonably manageable way attach different (post)processing pipelines to a single "source" depending on the event type (so that - for example - events from Palo Alto are processed differently than events from Fortigate while still being received on the same port). With rsyslog it's trivial, I'm not sure about syslog-ng.

3. There are two schools of thought regarding the "middle" between syslog receiver and Splunk. There is the old-school approach which you have in your config - receive events, write them to files, pick them up with monitor input, send to Splunk. There is also the modern-school approach - use http output/connector/whatever you call it to send it from your syslog receiver directly to HEC input.

Both approaches have their pros and cons. The file-based approach requires disk space for the files, requires manual cleaning of the directories but gives you a natural "buffer" if your downstream is not available (your Splunk is down). The HEC output approach is simpler in terms of infrastructure, lets you easily assign additional metadata but is more sensitive to downstream availability (and buffering is more complicated).

[0xAli]

Thank you @PickleRick  it's really appreciated!

I depends on the two the syslog-ng articles:

https://www.splunk.com/en_us/blog/tips-and-tricks/high-performance-syslogging-for-splunk-using-syslo... 

https://www.splunk.com/en_us/blog/tips-and-tricks/high-performance-syslogging-for-splunk-using-syslo...