Incorrect hostname for Cisco WAAS Logs

[lukemarrott]

I've got a weird issue with some Cisco WAAS devices identifying their hostname correctly in Splunk. We are in the process of migrating from an old solution to Splunk and previously got the correct hostnames for these.

I started using rsyslogd and it was breaking messages down into directories like this:
/var/log/syslog/%HOSTNAME%/%HOSTNAME%.log

Then Splunk was indexing these directories.

Well I ended up with some directories named '2012' and some directories named 'Apr'. Which are messages from our WAAS devices.

So to test it out I have disabled the indexing of this source and forwarded messages from rsyslogd to localhost on a different interface. Splunk still identifies the host according to part of the date. I have the option setup to set the source using DNS. And I have verified DNS lookup is functioning.

Any thoughts as to why these devices would not work and everything else seems to be fine?

Thanks!

[kristian_kolb]

Is it possible that you have two sets of timestamps in your log messages?

Am I wrong in guessing that your sourcetype for these events are 'syslog'? By default splunk will try to extract the host value from each event for this sourcetype. So there is a risk that anything that comes after the (first) timestamp will be interpreted as a hostname.

I'm guessing that your cisco devices put one timestamp in the event, and rsyslog adds another.

You can either;

a) reconfigure your cisco/rsyslogd so that you only get one timestamp per event
b) change your sourcetype to 'cisco_waas' or something that is NOT syslog, and specify host_segment=4 in the correct stanza in inputs.conf. This instructs Splunk extract the hostname from the path, where 4 would match %HOSTNAME%.

Hope this helps,

Kristian

[kristian_kolb]

You say that the hostname is lost. How is that?

Does %HOSTNAME% not expand to the actual hostname in rsyslog? checked the DNS?

Does rsyslog actually create the files correctly, i.e. with a real hostname under /var/log/syslog/xxx/xxx.log?

[lukemarrott]

So I have tried a few more things.

If I sent the message directly to Splunk then the hostname works fine. So it's something going on in rsyslogd that doesn't get the hostname from the WAAS. If I do a packet capture then the messages do include contain the source. But when it is going through rsyslogd either by processing to file or by being forwarded the hostname is somehow lost.

I am struggling with maintaining the original source IP/host on the forward. Trying to figure that out now.

Hmmmm...

Thanks. I'll look into this some more.