topic Re: Setting up Splunk monitoring of rsyslog with UDP in Getting Data In

Setting up Splunk monitoring of rsyslog with UDP

leongchongyu — Mon, 07 May 2018 03:42:45 GMT

I am running Splunk on an RHEL7 VM. I wish to be able to receive data from a Lexmark printer, which I have configured to send its logs through UDP to the VM's IP address on port 2500. I have also configured Splunk to monitor /opt/splunk/var/log/syslog/test.log for data through a data input. In my /etc/rsyslog.conf, I have the following configurations:

$ModLoad imudp
$UDPServerRun 2500
*.* /opt/splunk/var/log/syslog/test.log

However, when I trigger an event on the printer, it does not appear to log data to the test.log file. What could be the cause of this problem? I have tested the connectivity between the printer and the VM and it appears to be fine - for instance I can use a direct data connection to transfer data to Splunk. But for some reason it doesn't seem to able to send to the test.log file for Splunk to monitor.

Re: Setting up Splunk monitoring of rsyslog with UDP

xpac — Mon, 07 May 2018 16:41:40 GMT

Hey,

This obviously seems to be an syslog issue rather than a Splunk issue.

Two approaches:

use tcpdump to verify that data is actually received by your server, e.g. tcpdump -i eth0 udp port 2500, then send something and check if tcpdump looks fine (post it here if in doubt)
syslog-ng has proven to be easier to configure and to debug, so you might think about switching to it if you're just getting started with your syslog setup.

Hope that helps!

Re: Setting up Splunk monitoring of rsyslog with UDP

leongchongyu — Tue, 08 May 2018 01:51:31 GMT

Thank you for your prompt and informative response. I will switch to syslog-ng and try to use that instead.

Re: Setting up Splunk monitoring of rsyslog with UDP

FrankVl — Tue, 08 May 2018 08:02:26 GMT

Not much experience with syslog-ng, so can't confirm that would be easier. I think if you use the modern rsyslog syntax it is not massively different from syslog-ng. But basically your rsyslog config looks fine, so not too much of a point switching to another syslog daemon just to resolve this issue.

Indeed, start with using tcpdump to check if data is even coming in.

If so:
- confirm there is no firewall blocking it from reaching rsyslog (tcpdump sniffs before the firewall)
- confirm with netstat that rsyslog is indeed listening on that UDP port
- confirm rsyslog has permission to write to the indicated directory
- check /var/log/messages (or wherever rsyslog writes its own log) to check for rsyslog daemon errors.
- if there is nothing overly sensitive in it: post the entire rsyslog config file, just to check there is not something else in there that breaks your simple UDP input and file output config.

Re: Setting up Splunk monitoring of rsyslog with UDP

leongchongyu — Tue, 08 May 2018 08:28:52 GMT

Thank you so much! This was very helpful to me as I was struggling with setting up syslog-ng.

Re: Setting up Splunk monitoring of rsyslog with UDP

leongchongyu — Tue, 08 May 2018 08:34:02 GMT

# rsyslog configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

# The imjournal module bellow is now used as a message source instead of imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
#$ModLoad imklog # reads kernel messages (the same are read from journald)
#$ModLoad immark  # provides --MARK-- message capability

# Provides UDP syslog reception
$ModLoad imudp
$InputUDPServerRun 2500

# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 2048

#$InputTCPServer BindRuleset remote
#$TCPServerRun 2048





#### GLOBAL DIRECTIVES ####

# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

# Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
$OmitLocalLogging on

# File to store the position in the journal
$IMJournalStateFile imjournal.state

#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog


# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 :omusrmsg:*

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log

# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList   # run asynchronously
#$ActionResumeRetryCount -1    # infinite retries if host is down
# ### end of the forwarding rule ###

$template RemoteHost, "/opt/splunk/var/log/syslog/%HOSTNAME%/%FROMHOST-IP%.log"

*.* ?RemoteHost

This is my rsyslog.conf file. I changed it a bit as part of my trying to get it to work, so it's not exactly the same as it is above, so feel free to tell me if I screwed it up.

Re: Setting up Splunk monitoring of rsyslog with UDP

FrankVl — Tue, 08 May 2018 14:22:12 GMT

Took the liberty to put it in your post as code, such that it is actually readable.

One issue:

$InputUDPServerRun 2500

Should be (as you had it before, according to your question):

$UDPServerRun 2500

Am I overlooking something or is the *.* /opt/splunk/var/log/syslog/test.log part missing?

You replaced this by:

$template RemoteHost, "/opt/splunk/var/log/syslog/%HOSTNAME%/%FROMHOST-IP%.log"

 *.* ?RemoteHost

Which I think should work, but keeping it simple might make troubleshooting a bit easier.

But fix that UDPServerRun first and then check again.

What did you find for the other checks I suggested?

Re: Setting up Splunk monitoring of rsyslog with UDP

leongchongyu — Wed, 09 May 2018 02:52:41 GMT

1) Firewall isn't an issue. To the best of my knowledge it is not blocking anything.
2) The following are the results of running netstat -unl:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
udp        0      0 192.168.122.1:53        0.0.0.0:*
udp        0      0 0.0.0.0:67              0.0.0.0:*
udp        0      0 0.0.0.0:68              0.0.0.0:*
udp        0      0 0.0.0.0:50321           0.0.0.0:*
udp        0      0 0.0.0.0:1500            0.0.0.0:*
udp        0      0 0.0.0.0:18322           0.0.0.0:*
udp        0      0 0.0.0.0:5353            0.0.0.0:*
udp6       0      0 :::46355                :::*

3) To the best of my knowledge rsyslog has access to the directory and there are no restrictions on it.
4) Checking /var/log/messages comes up with errors for rsyslog, but none seem to pertain to the configuration I need. Some samples of messages recorded yesterday:

SELinux is preventing /usr/sbin/rsyslogd from append access on the file 127.0.0.1.log.
localhost python: SELinux is preventing /usr/sbin/rsyslogd from name_bind access on the tcp_socket port 9777
localhost python: SELinux is preventing /usr/sbin/rsyslogd from append access on the file /opt/splunk/var/log/syslog/localhost/127.0.0.1.log.

Nevertheless, if they're indicative of an issue, I can look up how to try to disable SELinux if you think it will help.

Most puzzlingly, I somehow seem to have some messages recorded in /var/log/messages that's dated May 10 - that is, one day in the future!

May 10 09:53:23 localhost rsyslogd-2212: imudp: module loaded, but no listeners defined - no input will be gathered [try http://www.rsyslog.com/e/2212 ]
May 10 11:10:47 localhost rsyslogd-2212: imudp: module loaded, but no listeners defined - no input will be gathered [try http://www.rsyslog.com/e/2212 ]

Needless to say I am very confused and unsure what to make of it.

Thank you so much for your extensive help - I'll try the configuration you suggested now.

Re: Setting up Splunk monitoring of rsyslog with UDP

FrankVl — Wed, 09 May 2018 07:08:50 GMT

So:

netstat says rsyslog is not listening on the port you wanted (probably because of that config mistake I pointed out alread)
/var/log/messages is showing an error that rsyslog cannot write (if I read the message correctly) to the file you specified due to some selinux intervention.

I'm not an selinux expert, so can't really give much specific suggestions for fixing that (apart from trying it with selinux disabled temporarily). You might want to try using a separate directory for writing those logs, rather than writing them to a subdir of your splunk installation. For instance create /opt/rsyslog/logs/ owned by the user under which rsyslog runs.