Syslog-ng, filter by ip

[fisk12]

I have some firewalls and stuff like that send logs to my Splunk server (using normal syslog at the moment). For now in "search" there are only one source "udp:514". I would like to filter so it lists all the different source (by ip, or by name i define for the source) I heard that you use syslog-ng for it. I have searched a bit but havt found any real good guides. Can someone push me in the right direction?

[southeringtonp]

It's worth noting that if you want to override only a few special cases, you may wish to use a transform instead. Take a look at:
http://answers.splunk.com/questions/5544/override-source-tcpxxxx-of-a-tcp-input-using-transforms

If you decide to keep Splunk as your listener, you may want to make sure that you have DNS resolution turned on, either through the Manager or by setting connection_host = dns in the stanza for port 514 in inputs.conf.

That out of the way, there are some other advantages to using syslog-ng, and many people consider that a best-practice. In particular, it gives you more freedom to restart Splunk without losing events.

---

Configuring for use with syslog-ng

What follows is a rough guide to setting it all up. Some options will vary depending on your OS and distribution.

By default, Splunk will name each source to the path of the syslog-ng logfile on your Splunk server, e.g., source=/inputs/syslog/myhost/messages.

You'll also want to take a look at the syslog-ng Administrators Guide for more detail on the syslog-ng configuration:
http://www.balabit.com/support/documentation/

Install syslog-ng

First, you'll need to actually install syslog-ng using the normal procedure for your OS (sudo yum install syslog-ng, or sudo apt-get install syslog-ng, or whatever).

You may need to uninstall or disable your existing syslog daemon before syslog-ng can be used.

Configure syslog-ng

Then, configure it to listen for incoming messages and save them to individual files based on where the message is coming from.

Note that this is not a complete configuration - you'll need to merge it with your default config file. Some additional tweaks, like flattening each section to a single line, may be needed if your distribution uses syslog-ng version 3.x.

# Misc options - use or leave out as desired.
options {
    # Resolve names from /etc/hosts, but don't do DNS lookups
    # Set to 'yes' if you want full DNS (slightly higher risk of DoS and delays)
    use_dns(persist-only);
    dns_cache_hosts(/etc/hosts);

    # Increase the maximum allowed length of an incoming message
    log_msg_size(8192);
}

# Start listening for messages on TCP and UDP ports
source s_remote {
    tcp(ip(0.0.0.0) port(514));
    udp(ip(0.0.0.0) port(514));
}


# Define a destination file or files
# Dumps all entries for that host into one file. Replace 'messages'
# with '$FACILITY' if you want to break out individual syslog facility.
destination d_split_by_host {
    file("/inputs/syslog/$HOST/messages"
        owner("root")
        group("splunk")
        perm(0640)
        dir_perm(0750)
        create_dirs(yes)
    );
}

# Route messages coming in via TCP or UDP sources to the right destination
log { source(s_remote); destination(d_split_by_host); flags(final); };

Disable Splunk listening on port 514

Turn off Splunk's listener on port 514, so that syslog-ng will be able to bind to the port.

In inputs.conf set:

[splunkudp://514]
disabled = true

or just delete the stanza for port 514 altogether.

Restart syslog-ng

Restart syslog-ng with /etc/init.d/syslog-ng restart or equivalent.

Verify that the syslog daemon is now listening on port 514 (sudo netstat -anp | grep 514 is one way).

Configure Splunk

Now, configure Splunk to monitor each directory. In inputs.conf:

[monitor:///inputs/syslog]
disabled = false
host_segment = 3
_blacklist = (.*\.gz$)|(lost+found)

By using host_segment, Splunk will automatically assign the host value based on the directory name, which syslog-ng will have created to match the IP address or hostname.

Override sourcetypes as needed

One of the disadvantages of this approach comes when Splunk fails to automatically recognize the sourcetype of log entries, especially for small files. You can override those as needed in props.conf:

[source::.../myfirewall/messages]
sourcetype=cisco_firewall

Set up Log Rotation

At this point, you should be able to start getting data into Splunk, but your log files will grow indefinitely. To prune them, if you are running logrotate, you can create a new file at /etc/logrotate.d/inputs-syslog:

/inputs/syslog/*/messages {
    daily
    minsize 10M
    rotate 1
    missingok
    sharedscripts
    postrotate
        /etc/rc.d/init.d/syslog-ng reload 2>/dev/null
    endscript
}

You'll need to add additional filenames (not just 'messages') for each syslog facility if you decide to use $FACILITY in syslog-ng's configuration.

[Starlette]

is it possible to check those configs on ng version 3?
Those are really different it seems!

[fisk12]

Thanks very much, much more detail in the answer then i excpected! Works like a charm.