topic Re: rsyslogd->forward into splunk via UDP - host always localhost(127.0.0.1) in Getting Data In

rsyslogd->forward into splunk via UDP - host always localhost(127.0.0.1)

besveinsson — Wed, 24 Sep 2014 12:31:01 GMT

So we are forwarding syslog using rsyslog to a udp port 2001 - all is working well except...

problem:

host is always 127.0.0.1

sample message looks like:
Sep 24 11:37:11 127.0.0.1 Sep 24 11:37:11 X.X.X.X 1693874: RP/0/RSP0/CPU0:Sep 24 11:37:11.073 GMT: tcp[395]: %IP-TCP-3-BADAUTH :

Where X.X.X.X is the IP of the sending syslog device.

host = 127.0.0.1 source udp:2001 sourcetype = syslog

Is it possible to get those IP's into the host tag - as everything is tagged to 127.0.0.1 ??

I have looked at some answers pointing to editing transforms.conf and props.conf (I edited the /opt/splunk/system/local files)
but nothing is working

I also get the double timestamps - both when Splunk receives the message and also the Cisco timestamp.

any ideas ?

Re: rsyslogd->forward into splunk via UDP - host always localhost(127.0.0.1)

srioux — Wed, 24 Sep 2014 13:18:29 GMT

Without seeing your configs or having access to the environment, I would note that it's likely an issue with rsyslogd output rather than Splunk input. It looks like it's tagging itself as part of the syslog message chain. Have you tried updating the templates for log messages in the rsyslog config (typically /etc/rsyslog.conf)? Docs (for v5, not sure what version you'd be running):
http://www.rsyslog.com/doc/v5-stable/configuration/templates.html

As reference, here's a sample template we've used for some of our syslog events:

# Create a template to prevent double timestamps
$template juniper,"%timestamp:::date-rfc3339% %HOSTNAME%%msg%\n"

Not that it's directly related to your problem... but from an architectural point of view, I'd recommend dropping those events to a local log file rather than having rsyslog send directly to Splunk. That way, you have additional resiliency in case you need to take Splunk down (ex: upgrades). Just have Splunk monitor those particular log file(s). If you go this route, also make sure to add logrotate configs for those files too.

Re: rsyslogd->forward into splunk via UDP - host always localhost(127.0.0.1)

besveinsson — Wed, 24 Sep 2014 13:29:46 GMT

ok - I will look at that -

I'm not quite new to spunk - but in this installation we're using rsyslogd - but I have used syslog-ng in the past. I used to be able to point spunk to the directory and it just grabbed all log files recursively. In this case, spunk will not match the log files. We use log-rotate and .gz old files.

logs from each host are put in subdirectories - files are named by the date .log

I'm probably missing some basic stuff here.

going to look at rsyslog conf and further how to get those files into spunk (better!)

Benni

Re: rsyslogd->forward into splunk via UDP - host always localhost(127.0.0.1)

srioux — Wed, 24 Sep 2014 18:20:59 GMT

Another note:
Might also be worth checking the UDP inputs on inputs.conf
http://docs.splunk.com/Documentation/Splunk/6.1.3/admin/Inputsconf

no_appending_timestamp = [true|false]
* If this attribute is set to true, Splunk does NOT append a timestamp and host to received events.
* NOTE: Do NOT include this attribute if you want to append timestamp and host to received events.
* Default is false.

Re: rsyslogd->forward into splunk via UDP - host always localhost(127.0.0.1)

vqd361 — Thu, 25 Sep 2014 05:59:54 GMT

What version of rsyslog are you using?