Getting Data In

Incorrect hostname for Cisco WAAS Logs

lukemarrott
Engager

I've got a weird issue with some Cisco WAAS devices identifying their hostname correctly in Splunk. We are in the process of migrating from an old solution to Splunk and previously got the correct hostnames for these.

I started using rsyslogd and it was breaking messages down into directories like this:
/var/log/syslog/%HOSTNAME%/%HOSTNAME%.log

Then Splunk was indexing these directories.

Well I ended up with some directories named '2012' and some directories named 'Apr'. Which are messages from our WAAS devices.

So to test it out I have disabled the indexing of this source and forwarded messages from rsyslogd to localhost on a different interface. Splunk still identifies the host according to part of the date. I have the option setup to set the source using DNS. And I have verified DNS lookup is functioning.

Any thoughts as to why these devices would not work and everything else seems to be fine?

Thanks!

0 Karma

kristian_kolb
Ultra Champion

Is it possible that you have two sets of timestamps in your log messages?

Am I wrong in guessing that your sourcetype for these events are 'syslog'? By default splunk will try to extract the host value from each event for this sourcetype. So there is a risk that anything that comes after the (first) timestamp will be interpreted as a hostname.

I'm guessing that your cisco devices put one timestamp in the event, and rsyslog adds another.

You can either;

a) reconfigure your cisco/rsyslogd so that you only get one timestamp per event
b) change your sourcetype to 'cisco_waas' or something that is NOT syslog, and specify host_segment=4 in the correct stanza in inputs.conf. This instructs Splunk extract the hostname from the path, where 4 would match %HOSTNAME%.

Hope this helps,

Kristian

0 Karma

kristian_kolb
Ultra Champion

You say that the hostname is lost. How is that?

Does %HOSTNAME% not expand to the actual hostname in rsyslog? checked the DNS?

Does rsyslog actually create the files correctly, i.e. with a real hostname under /var/log/syslog/xxx/xxx.log?

0 Karma

lukemarrott
Engager

So I have tried a few more things.

If I sent the message directly to Splunk then the hostname works fine. So it's something going on in rsyslogd that doesn't get the hostname from the WAAS. If I do a packet capture then the messages do include contain the source. But when it is going through rsyslogd either by processing to file or by being forwarded the hostname is somehow lost.

I am struggling with maintaining the original source IP/host on the forward. Trying to figure that out now.

Hmmmm...

Thanks. I'll look into this some more.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...