Getting Data In

Howto Setup RSYSLOG network event log filtering based on message type - content (sourcetype)

milesbrennan
Path Finder

I've written this RSysLog configuration file, which allows network based devices to send their event logging streams to a centralised RSysLog server(s), and the each incoming event will be placed into a specific log file, depending on the contents of the received message. This in effect should allow splunkers to identify which log file holds Cisco ASA events, and which log file holds CheckPoint events - inturn, this will ease setting sourcetypes on network based devices. Syslog server just needs a universal forwarder to send logs to indexers, and an inputs.conf file matching each filename to each sourcetype.

For example, all Cisco ASA firewall events contain the text "%ASA-" and should go into "cisco-firewall.log" file, and Cisco SourceFire uses a "$programname" of "SFIMS" in the message, so these will be forwarded to "cisco-sourcefire.log", etc... Any inbound network event which does not match any of the filtering rules, will be sent to "uncategorised.log" in message debug mode, so they can be assessed, and a new filter written for new message type.

So there's nothing wrong with this RSysLog configuration, it works perfectly, but I spent a fair amount of time building and testing it with the new configuration commands, I'm sure there are a few people here who would also benefit from it's use. Additionally, collecting logs on a low maintenance Linux server, allows you to do more changes / reboots of the Splunk indexers, that you don't need to worry about missing network event logs if you have to restart an indexer server / service - RSysLog will just keep doing its job.

Further details on RSysLog properties can be found here: http://www.rsyslog.com/doc/master/configuration/properties.html

This setup opens listeners on UDP514 and TCP10514 - You can change these to suit your own requirements.

Regards.

#
#     File:     "splunk.conf"
#     Location: "/etc/rsyslog.d/"
#     Author:   Miles Brennan
#     Date:     31 Dec 2015
#     Ver:      0.3
#


#
#    Prerequisites:
#    Splunk is running as Linux user:      adduser splunk
#    Add "splunk" user to syslog groups:   usermod -G sys,adm,syslog splunk
#    Create logging directory for Splunk:  mkdir /var/log/splunk
#    Set permissions for log directory:    chown -R syslog:adm /var/log/splunk /home/splunk
#

#
#    Splunk Universal Forwarder - Input Configuration (On Deployment Server)
#    File: cisco-firewall.log           Sourcetype: Cisco ASA Firewalls
#    File: cisco-sourcefire.log         Sourcetype: Cisco SourceFire
#    File: checkpoint-firewall.log      Sourcetype: Checkpoint Firewalls
#


#    Set up log file rotation and compression for the new RSyslog files.
#
# vi /etc/logrotate.d/splunk


#/var/log/splunk/*.log {
#        daily
#        missingok
#        rotate 14
#        compress
#        notifempty
#        nocreate
#}


# Load the RSyslog modules needed for inbound connections
#
module (load="imudp")
module (load="imtcp")

# Define the RSysLog logging format for writing events to syslog.
template (name="rsyslog-fmt" type="string"
       string="%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
)

# Review all incoming message streams and deliver message to
# appropriate log file, as per RegEx.
#
# Save in new logging directory at: /var/log/splunk
#
ruleset (name="network-logs") {
if $msg contains "%ASA-" then            { action (type="omfile" file="/var/log/splunk/cisco-firewall.log"         template="rsyslog-fmt") stop }
if $programname contains "SFIMS" then    { action (type="omfile" file="/var/log/splunk/cisco-sourcefire.log"       template="rsyslog-fmt") stop }
if $msg contains "Check Point" then      { action (type="omfile" file="/var/log/splunk/checkpoint-firewall.log"    template="rsyslog-fmt") stop }

# Catch the remaining log events which passed through our filters.
action (type="omfile" file="/var/log/splunk/uncategorised.log" template="RSYSLOG_DebugFormat") stop
}

# Bind the UDP module for inbound connections on port 514,
# and apply "remote" ruleset to all incoming messages.
#
input (type="imudp" port="514" ruleset="network-logs")

# Bind the TCP module for inbound connections on port 10514,
# and apply "remote" ruleset to all incoming messages.
#
input (type="imtcp" port="10514" ruleset="network-logs")
1 Solution

milesbrennan
Path Finder

Hi jkat54, you are correct, my post is not a question, it is actually a guide for people to follow and use if they need different options to centralise logs before ingesting them into Splunk.

We use RSysLog servers to centralise a lot of our network device logs and filter them into specific file names based on what their role / function is, then we have a small application deployed to the universal forwarder, which collects the logs and assigns the appropriate sourcetypes. i.e. Cisco ASA firewall logs will be assigned cisco:asa

Using RSysLog to collect our logs means the indexers can be taken offline for maintenance or rebooted without data loss (for UDP traffic), we can also set up the universal forwarder for round robin delivery to multiple indexers (better performance), and if another department in our organisation also needs access to the network log files (and they don't have, or use Splunk).. then the log files will still be stored on the RSysLog server for a few weeks for them to get the files... you don't want to start pulling log files out of Splunk for a completely different purpose.

So because of the many use cases I needed, I decided to make this availabe for others to use, if they have complex configurations and are looking are different solutions, then they only need to copy this configuration into RSysLog and it will work.

This was written and validated on RSyslog 8.15.0 using the newer configuration standards and options.

View solution in original post

jaydee12
Engager

@milesbrennan - Thanks for the post, very helpful.  Have a question around parsing/formatting logs so that they are ingested in splunk with appropriate fields.  Is it common practice to do this in rsyslog before being sent to Splunk?  In my test setup, I have a Rsyslog server with the universal forwarder and then sending them to my splunk instance.  I am new to splunk and logging, so hope the question makes sense.

0 Karma

willsy
Communicator

may be years old but you my good sir are both a scholar and a gent. much appreciated

0 Karma

milesbrennan
Path Finder

Use the following commands to create the /var/log/splunk directory, then all the user, group permissions will flow through into the directory, and set the correct ACLs on newly created files - automatically.

mkdir /var/log/splunk
chown root.splunk /var/log/splunk
chmod 755 /var/log/splunk
chmod g+s /var/log/splunk
setfacl -d -m u::rwX,g::rX,o::- /var/log/splunk
service rsyslog restart

The above allocates "splunk" as the group for both directory and files, and allows read access to splunk user. RSysLog still runs as root, and logrotate functions as expected.

0 Karma

lakshman239
SplunkTrust
SplunkTrust

Thanks Miles. That helps. creating empty files with 'splunkusr', so rsyslog running as root doesn't change it.

0 Karma

lakshman239
SplunkTrust
SplunkTrust

Thanks Miles. It works on rsyslogd 7.4.10 on RHEL 6.7
For now, I have setup 514/UDP and verified it using netstat -an | grep 514 and then used netcat to send test message to ensure 'uncategorised.log' is being written. I'll test the cisco ASA later.[ http://mikeberggren.com/post/53883822425/ncudp]

0 Karma

milesbrennan
Path Finder

Hi jkat54, you are correct, my post is not a question, it is actually a guide for people to follow and use if they need different options to centralise logs before ingesting them into Splunk.

We use RSysLog servers to centralise a lot of our network device logs and filter them into specific file names based on what their role / function is, then we have a small application deployed to the universal forwarder, which collects the logs and assigns the appropriate sourcetypes. i.e. Cisco ASA firewall logs will be assigned cisco:asa

Using RSysLog to collect our logs means the indexers can be taken offline for maintenance or rebooted without data loss (for UDP traffic), we can also set up the universal forwarder for round robin delivery to multiple indexers (better performance), and if another department in our organisation also needs access to the network log files (and they don't have, or use Splunk).. then the log files will still be stored on the RSysLog server for a few weeks for them to get the files... you don't want to start pulling log files out of Splunk for a completely different purpose.

So because of the many use cases I needed, I decided to make this availabe for others to use, if they have complex configurations and are looking are different solutions, then they only need to copy this configuration into RSysLog and it will work.

This was written and validated on RSyslog 8.15.0 using the newer configuration standards and options.

jkat54
SplunkTrust
SplunkTrust

@milesbrennan

Please mark your answer as the answer to the question so this "question" doesnt appear to not have an answer

0 Karma

lakshman239
SplunkTrust
SplunkTrust

The uncategorised.log file has been created as 'root' user and 'root' group, possibly as rsyslog is running as root. How do I tell it to write using splunkusr:splunkgroup.
I tried $FileOwner and $FileGroup, but still the file is owned by root. any thoughts?

0 Karma

milesbrennan
Path Finder

Each Linux distro is a little different. You can create the file then set the permissions manually (which you should only need to do on initial setup)

touch /var/log/splunk/uncategorised.log
chown -R syslog:adm /var/log/splunk
service rsyslog restart (or equivilant)

If you're running as a different user, you will need to create each new category file and assign permissions manually (as above) after setting up each new template. Alternately, you can add the "splunk" user into the group which runs syslog (see "/etc/groups"). Note, the splunk user only needs read access to logs, leave RSysLog to manage files.

FYI - We're using Ubuntu 8.4 64bit for the syslog server.

0 Karma

lakshman239
SplunkTrust
SplunkTrust

Thanks Miles. On servers where rsyslog is running as root, we don't need to run chown command, as it can read the files. correct?

will this still work in rsyslog7 ?

0 Karma

sobrien
Splunk Employee
Splunk Employee

Thanks for sharing, Miles!

0 Karma

jkat54
SplunkTrust
SplunkTrust

You question is how to setup RSYSLOG, but then you answered your question... so what is your SPLUNK question?

How to monitor the files created by syslog?

If so, see this:
http://docs.splunk.com/Documentation/Splunk/6.2.0/Data/Configureyourinputs

if not, please clearly state your SPLUNK question or take any of your SYSLOG questions to another forum.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...