I've set up Snare on remote servers to forward syslog events on port 6161 to my Splunk server.
I've run wireshark on my Splunk server to ensure the events are arriving -- they are.
I've added a UDP data source on port 6161 to accept the syslog messages.
After 24 hours, there's nothing showing up, even though I've confirmed they are still arriving via wireshark.
Restarting the service has not resolved either. Ideas?
It sounds like the events are not being identified with the right search, or the input is not working correctly. I recommend:
1 - verify that you can locally send udp traffic to port 6161. On unix systems, using netcat or similar is a good test
2 - Run a search for
"index=* source=udp:6161" over all time
3 - Run a metrics search in the internal index to see if you have data incoming from that port. For example:
"index=_internal source=*metrics.log* per_source_thruput series=udp:6161"
Check your iptables rules. Wireshark is great, but its capture modules plug into the network stack below iptables -- so wireshark will see packets arrive at the interface that iptables could deny/drop after the fact.
The same advice for iptables rules applies to the Windows firewall as well.
If it's really UDP you can't telnet to it. Telnet does not work with UDP, it uses TCP.
Sorry, it was supposed to say "the local firewall is off". There is no firewall service running on the system.
Just for fun, I enabled the FW, added a UDP 6161 allow rule, and re-disabled it.
A few more things to try:
Double-check your inputs configuration, and see if you are overriding `source` or `sourcetype` -- if you are, modify the searches suggested by Simeon accordingly.
Broaden your searches and try to match on raw text instead of source or sourcetype to see if those fields have something different than expected. Pick a string of text that you know appears in your Windows logs, and search on that: `index=* SomeRawText`
Run `netstat -p udp -b` to verify that the Splunk daemon is actually the process bound to that port.
If you suspect a firewall issue, try stopping Splunk and installing Kiwi Syslog or similar on that port, just long enough to verify that it can receive syslog messages. That will help narrow down the problem to either network/host configuration or Splunk configuration.