Netapp and Syslog data

[eegilbert]

I've setup a data input for syslog on both TCP and UDP 514. Pretty straight forward and I've verified I am getting logs into splunk from actual syslog servers, however I'm having a interesting problem with netapp devices.

The netapp devices are configured to send their syslog directly to the IP address of the splunk server. I can see the messages coming in via UDP if I run TCP dump however these never show up in a search.

In fact the only way I can get these messages to show up is to delete the UDP data input and setup an actual syslog daemon for UDP/514 and then log to a file, which then splunk picks up.

There is no firewall or acl blocking the syslog access to the splunk box so I'm at a loss as to why these messages are having so much trouble.

Anyone see this? Anyone else using splunk for monitoring of netapp data?

Thank you,

Erric

[tfadmin]

I was noticing something similar in setting up our filer to connect to SPLUNK. You may want to check what user SPLUNK is running as. I have it running as splunk, so it cannot bind to 514/UDP. I have to use a IPTABLES forwarding rule to forward from 514 to a port that SPLUNK is allowed to bind to:

Sample from /etc/sysconfig/iptables
-A PREROUTING -p udp -m udp --dport 514 -j REDIRECT --to-ports 5447

Then Splunk is actually setup to listen on 5447

BTW - On the NETAPP, I was able to use
. @IP_ADDRESS
(make sure the spaces are actually a tab. apparently can be can issue)

Cheers

[skippylou]

I bet your log data is actually in there when sending via UDP directly from your Netapps, however, I imagine it is your host field of the Netapp that is skewed.

Have a look at the logs first that you are writing to disk from it via the syslog to a file, I'm guessing if they are like mine they look like:

date time hostname [hostname: process:priority]: message

That first 'hostname' is getting written by the intermediary syslog server you are sending logs to first, which when you have splunk read this file picks up the host filed properly.

What I would do to see if the direct from Netapp logs are in splunk is to instead of putting in your search:

 host=mynetappname

which I am guessing you are doing, do instead:

host=*mynetappname*

I'm betting it is the host field at index time that is getting munged when you are sending direct. Netapp for whatever reason jumbles the hostname in with the process and syslog priority and I bet that is causing issues here - at least from my experience with their logs.

Hope this helps,

Scott