Getting Data In

Example - rsyslog.conf for multiple data sources with UF

cbutler8329
New Member

RHEL 5.9 with rsyslog 3.22
Splunk 5.0.2
Universal Forwarder installed, with the intention of monitoring logs processed by rsyslog.

I have little to no knowledge of rsyslog. I have searched splunk-base extensively for example configurations.

We have 5-6 data sources coming in on two different UDP ports.

514 contains 4-5 of these data sources (Cisco FWSM, DNS, routers, swithes, etc)
516 contains palo alto.

The log sources go to an log repeater, which we can forward on any port to the rsyslog/UF. So, going with high ports.
10514
10515
10516, etc

Need a rsyslog.conf example with filters to break out the 514 data sources into directories by hostname. Will use log rotate to clean up after ingest by UF. Have 500GB coming in daily, so we can only keep 12 hours or so on the rsyslog server for "buffer".

We are running into issues with the older style selector/rule contexts. Everyone seems to have switched to the new context in the rsyslog.conf file, which I am not entirely sure is supported in rsyslog v3.22.

I also notice everytime I start rsyslog it runs the -c 5 option in for backwards compatibility. It yells at me to use -c3 to eliminate backwards compatibility due to that causing other issues. I wonder if that is part of the problem. I have manually run it with -c3 and it cleans up the errors in the logs, but no joy on filtering and breaking out the configs to the degree I am looking for.

The config below does work, but I don't have "matching" sample data to test with logger or nc. However,what I do test with (from Mac or RHEL /var/log files) using nc and logger does break it out by host, but it also copies it to /var/log/messages even when connecting with "nc -u 192.168.56.50 10514".

Appreciate any help or pointers to other answers. Looking for an actual real world "working" rsyslog.conf and matching UF inputs.conf. I seem to learn by example 🙂

syslog-ng is not an option, as it is not approved software.


Provides kernel logging support (previously done by rklogd)

$ModLoad imklog

Provides support for local system logging (e.g. via logger command)

$ModLoad imuxsock

--------------------------------------------------------------------------------

Provides UDP syslog reception

$ModLoad imudp.so
$UDPServerRun 10514

$UDPServerRun 10515

$UDPServerRun 10516

$UDPServerRun 10517

$UDPServerRun 10518

--------------------------------------------------------------------------------

GLOBAL DIRECTIVES

$umask 0000
$DirCreateMode 0775
$FileCreateMode 0640
$FileOwner root
$FileGroup root
$DirOwner root
$DirGroup root

--------------------------------------------------------------------------------

Use traditional timestamp format

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

--------------------------------------------------------------------------------

Log all kernel messages to the console.

Logging much else clutters up the screen.

kern.* /dev/console

Log anything (except mail) of level info or higher.

Don't log private authentication messages!

*.info;mail.none;authpriv.none;cron.none /var/log/messages

The authpriv file has restricted access.

authpriv.* /var/log/secure

Log all the mail messages in one place.

mail.* -/var/log/maillog

Log cron stuff

cron.* /var/log/cron

Everybody gets emergency messages

*.emerg *

Save news errors of level crit and higher in a special file.

uucp,news.crit /var/log/spooler

Save boot messages also to boot.log

local7.* /var/log/boot.log

$template DailyPerHostLogs,"/opt/netlogs/%HOSTNAME%/%HOSTNAME%-%$YEAR%%$MONTH%%$DAY%%$HOUR%.log"
. -?DailyPerHostLogs

0 Karma

datasearchninja
Communicator

rsyslog 3.22 is not going to support -c 5. It only supports compatibility up to -c 3.

RHEL 5.9 did ship with a rsyslog 5 version in package rsyslog5. If you can upgrade to that I would, as it is much more logical to run in the configuration file.

In rsyslog5, then you can split the config out using rulesets, and even split out based on that incoming port:

# rsyslog v5 configuration file

$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imklog   # provides kernel logging support (previously done by rklogd)

# Provides UDP syslog reception
$ModLoad imudp

$ActionFileDefaultTemplate RSYSLOG_FileFormat

$RuleSet local

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog

# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 *

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log

$DefaultRuleset local


$template Remote10514,"/opt/netlogs/10514/%HOSTNAME%/%HOSTNAME%-%$YEAR%%$MONTH%%$DAY%%$HOUR%.log"

$RuleSet Remote10514
*.* -?Remote10514

$InputUDPServerBindRuleset Remote10514
$UDPServerRun 10514


$template Remote10515,"/opt/netlogs/10515/%HOSTNAME%/%HOSTNAME%-%$YEAR%%$MONTH%%$DAY%%$HOUR%.log"

$RuleSet Remote10515
*.* -?Remote10515

$InputUDPServerBindRuleset Remote10515
$UDPServerRun 10515

etc

If you must run rsyslog v3, then the key is to process messages in an order, and use '& ~' to stop processing messages after all are processed. The next stanza can then process based on what is left.

So the idea might be to send everything that didn't originate from the local host to the /opt/netlogs/ file path, then stop processing using '& ~'. The messages that are left must be from the local host, so you can let fall out to the /var/log/messages config. See http://www.rsyslog.com/storing-messages-from-a-remote-system-into-a-specific-file/

0 Karma

cbutler8329
New Member

Colin, I checked the repo, and rsyslog5 is there. Checking on upgrade options.

we have 5 or 6 data sources combined into a single UDP 10514. I don't claim to know rsyslog, but . assumes that is a single data source in the stream correct?

Do you have working filter examples if we have well defined FQDN in the flows?

For exmaple...all DNS servers contain "xyzw" in the name, which can be directed to its own separate directory

Then on the same 10514 I have to filter out the remaining data sources after extracting the 34 DNS servers.

Thanks..Chris

0 Karma

cbutler8329
New Member

Colin...thanks for the response. I failed to mention two things. I was already going down the path of rulesets...very cool concept. But it didn't seem to work. Assuming because it was v3.22 rsyslog.

The second thing I failed to mention is...I am not allowed to upgrade to v5.

I am forced to use v3 rsyslog, or start listening to these ports on the splunk forwarder, which I really really do not want to do. I want to pick up the files via a monitor after rsyslog drops them.

I saw the & ~ stuff. DIdn't try that, because the rulesets looked cleaner/nicer. 🙂

Thanks much,

Chris

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...