Splunk not receiving syslog events, but telnet works

[bclapp]

I've got splunk working properly on a CentOS 6.5 box. I have another CentOS box client, and I can telnet to the port 8001 on the splunk box and send events, and they show up in splunk. But rsyslog events don't ever show up in splunk.

Here's the only modification that I made to the end of /etc/rsyslog.conf:

# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
*.* @@splunk.mydomain.com:8001

And here's what happens when I telnet to it:

[root@app ~]# telnet splunk.mydomain.com 8001
Trying 10.1.3.203...
Connected to splunk.mydomain.com.
Escape character is '^]'.
hi
^]
telnet> quit
Connection closed.

And that results in a "hi" event showing up in splunk.

Now, if I log something from the command line, it shows up in /var/log/messages:

[root@app ~]# tail /var/log/messages -n 1
Mar 12 16:00:13 app test: WHATEVER BRO

But I get nothing new in splunk, and a search for "WHATEVER BRO" turns up nothing... What am I missing?

Thanks!

[toby_meyer]

Hey I'm sure you moved on to the forwarder (as one should noted by starcher), but if anyone else stumbles upon this I thought I should post up the answer.

This is likely a SELinux problem; it will block outgoing connections on "non-standard" ports on a per-server basis. To fix, you'll need to add your custom port to the service definition for syslog (rsyslog is launched as syslog for policy purposes) or disable SELinux. To test without rebooting, you can try the command setenforce 0 (not persistent across reboots). If it works after that, there's your problem. I might write up a tutorial post later, if so I'll circle back and link it.

That said, I'm moving on to the forwarder as starcher suggests; just couldn't let this one go due to its odd behavior!

[starcher]

can you list what the setup of the network port on the Splunk side is that you setup? However is it normally not a good idea to send syslog straight to splunk. Use the universal forwarder on your linux system running rsyslog. Have rsyslog write logs to folders and use the universal forwarder to pick up those folders and send to splunk. This lets you choose different indexes and source types depending on what the log data actually is based on each source being collected.