topic Re: Getting Splunk to work with Syslog UDP 514 in Getting Data In

Getting Splunk to work with Syslog UDP 514

fletch13 — Thu, 10 Oct 2013 21:16:50 GMT

I've been reading and trying to figure this out. But i'm stomped.

I configured a device to send syslog events to the splunk server via udp:514
i can see the traffic (on splunk server)with tcpdump port 514. I've tested this by trigger an event on the device and seeing the event on the splunk server (via tcpdump.

./splunk list udp cli shows 514
I've tailed "metics.log" and i do not see the related syslog event getting there.

Looks like everything is good. but nothing in searches or indexes..
running splunk ver 6.

Re: Getting Splunk to work with Syslog UDP 514

lukejadamec — Thu, 10 Oct 2013 22:00:43 GMT

How have you configured Splunk to listen on that port?

Re: Getting Splunk to work with Syslog UDP 514

dwaddle — Fri, 11 Oct 2013 05:32:17 GMT

Disable your firewall. And/or add rules to allow UDP/514.

Using tcpdump is a great test, but it is misleading with UDP packets. The libpcap libraries sit in the network stack below iptables. So it is entirely possible that you will see a packet arrive with tcpdump and it will be dropped by iptables before it makes it to the process. And because it's UDP there is no broken session setup to observe.

Re: Getting Splunk to work with Syslog UDP 514

twinspop — Fri, 11 Oct 2013 17:17:55 GMT

Dwaddle nailed it, but I'd also check to see if another process, like syslog or syslog-ng, is already using UDP 514.

lsof -i :514

Re: Getting Splunk to work with Syslog UDP 514

fletch13 — Fri, 11 Oct 2013 17:38:12 GMT

yes. Splunk has been configured to listen to udp 514

Re: Getting Splunk to work with Syslog UDP 514

fletch13 — Fri, 11 Oct 2013 17:43:22 GMT

okay. i double checked and iptables files is configured to accept port 514. plus restarted iptables.
-A INPUT -p udp -m udp --dport 514 -j ACCEPT

Re: Getting Splunk to work with Syslog UDP 514

fletch13 — Fri, 11 Oct 2013 17:51:38 GMT

i did have rsyslog running. i kill it and still not seeing my udp traffic in the metrics.log

Re: Getting Splunk to work with Syslog UDP 514

twinspop — Fri, 11 Oct 2013 20:25:20 GMT

Did you restart Splunk after? It won't dynamically re-try the port. A restart will be needed.

Re: Getting Splunk to work with Syslog UDP 514

fletch13 — Fri, 11 Oct 2013 22:26:45 GMT

Not sure but it works now. Could have been just going over steps needs to make this work.
thanks

configured iptables to take udp 514
restarting iptables
adding data input udp 514

BTW: i still don't see syslog "port 514" activity in the metrics.log but it works.

Re: Getting Splunk to work with Syslog UDP 514

mcronkrite — Wed, 02 Jul 2014 01:02:10 GMT

Redirects 514 to 5514 in this example.

Poke hole in iptables to allow web-configuration and the listener ports

Redirect for port 515 up to 5514 which we are listening on (be sure to "service iptables save" after modifying iptables, or modify etc/sysconfig/iptables directly)

iptables -I INPUT -p tcp --dport 8000 -j ACCEPT

iptables -I INPUT -p tcp --dport 5514 -j ACCEPT

iptables -t nat -A PREROUTING -d MY.IP -p tcp -m tcp --dport 514 -j DNAT --to-destination MY.IP:5514

iptables -t nat -A PREROUTING -d MY.IP -p udp -m udp --dport 514 -j DNAT --to-destination MY.IP:5514