Getting Splunk to work with Syslog UDP 514

[fletch13]

I've been reading and trying to figure this out. But i'm stomped.

I configured a device to send syslog events to the splunk server via udp:514
i can see the traffic (on splunk server)with tcpdump port 514. I've tested this by trigger an event on the device and seeing the event on the splunk server (via tcpdump.

./splunk list udp cli shows 514
I've tailed "metics.log" and i do not see the related syslog event getting there.

Looks like everything is good. but nothing in searches or indexes..
running splunk ver 6.

[mcronkrite]

Redirects 514 to 5514 in this example.

Poke hole in iptables to allow web-configuration and the listener ports

Redirect for port 515 up to 5514 which we are listening on (be sure to "service iptables save" after modifying iptables, or modify etc/sysconfig/iptables directly)

iptables -I INPUT -p tcp --dport 8000 -j ACCEPT

iptables -I INPUT -p tcp --dport 5514 -j ACCEPT

iptables -t nat -A PREROUTING -d MY.IP -p tcp -m tcp --dport 514 -j DNAT --to-destination MY.IP:5514

iptables -t nat -A PREROUTING -d MY.IP -p udp -m udp --dport 514 -j DNAT --to-destination MY.IP:5514

[fletch13]

Not sure but it works now. Could have been just going over steps needs to make this work.
thanks

1. configured iptables to take udp 514
2. restarting iptables
3. adding data input udp 514

BTW: i still don't see syslog "port 514" activity in the metrics.log but it works.

[fletch13]

yes. Splunk has been configured to listen to udp 514

[twinspop]

Dwaddle nailed it, but I'd also check to see if another process, like syslog or syslog-ng, is already using UDP 514.

lsof -i :514

[twinspop]

Did you restart Splunk after? It won't dynamically re-try the port. A restart will be needed.

[fletch13]

i did have rsyslog running. i kill it and still not seeing my udp traffic in the metrics.log

[dwaddle]

Disable your firewall. And/or add rules to allow UDP/514.

Using tcpdump is a great test, but it is misleading with UDP packets. The libpcap libraries sit in the network stack below iptables. So it is entirely possible that you will see a packet arrive with tcpdump and it will be dropped by iptables before it makes it to the process. And because it's UDP there is no broken session setup to observe.

[fletch13]

okay. i double checked and iptables files is configured to accept port 514. plus restarted iptables.
-A INPUT -p udp -m udp --dport 514 -j ACCEPT

[lukejadamec]

How have you configured Splunk to listen on that port?