CentOS 4 to forward syslog to Splunk Indexer but no data was forwarded after configuring

[Kitteh]

I have already appended my Splunk IP Address and UDP port in /etc/syslog.conf "(asterisk).(asterisk) (asterisk)192.168.0.1/9995", restarted syslog service too.

At the Splunk side, I also added a new data input UDP, to accept 9995 and restrict the host to only the CentOS machine which is 192.168.0.59 (ip of my centos) but to no avail, no data was sent to Splunk...

There is connection between two, checked via ping.

[Richfez]

I'll have some things to check later, but first - can you just install the Universal Forwarder on your CentOS box? That will be far more reliable than using syslog straight to Splunk as a TCP input - the UF will queue data up when Splunk restarts instead of just dropping packets, is better performing, gives better quality data (better meta information), and gives a LOT more options for what data you can collect. I would recommend this if it's even barely possible. It really will be better. 🙂

BUT, for your actual problem...

1) Try a real time search first searching the index you have the UDP input configured to send to, like index=X for a 5 minute window. Wait a bit or do something on the centos box to make a syslog entry show up in its own logs, see if it shows up in Splunk. If data comes in here, double-check that data seems to have the right date/time on them. Sometimes this gets messed up and while the data actually does show up, it's not "visible" easily becuase it's in the future or from 1970 so a "last 4 hours" search won't show it. We can fix this if this is the problem, but it's not worth trying to fix unless it IS the problem. 🙂

2) If no data came in on step #1, run your flavor of packet sniffer (tcpdump, wireshark, whatever) on the Splunk machine and look for data from your CentOS host to be coming on on the configured port 9998. If this doesn't work, you have a mistake in the syslog configuration. Please double-check that. This could also be a firewall problem (just because you can ping it doesn't mean you have the actual 9998 port open!). You can check for local firewalls on either system and temporarily disable them (obviously, this is testing only and adjust what you actually do to fit your own security needs and stance!), and even double-check any hardware/network firewalls between the two.

3) If data doesn't come in (e.g. #1 has no data) but network traffic is found, you have a misconfiguration in Splunk. Either the UDP input is "doing something weird" to the data (unlikely, but if you paste in the inputs from your inputs.conf for this it would help us determine is something is goofy there - please be sure to paste it using the little 101010 "code" button from the toolbar!), or it's being sent to the wrong index or something.

If all that seems OK, then please a) include the inputs.conf stanzas for this inputs (if that's gibberish to you, please describe more fully all the little parts of the input - index and things are important), and b) let us know exactly What searches you have run that tells you you "aren't seeing data in Splunk" so we can help you if it's just a bad search.

Happy Splunking,
-Rich