I have a UDP/514 Port setup in data inputs. i have a number of machines sending syslog data to this port however only certain applications show up in splunk. the rest never get there. Is there any kind of log that would show what is happening? is splunk dropping the data. I see on wireshark that the data leaves the sending machine and i am aware that UDP does not ensure delivery but i would expect at least one or two packets to make it?
Any ideas why this would be?
Please check 3 things (if you are on linux)
I have the same problem. Firewall on Splunk server is disabled, changed rp_filter to 0, I see the packets from both Cisco firewalls in tcpdump, but only see events from one firewall being indexed.
Have you run a tcpdump/wireshark on the machine running Splunk?
Have you confirmed the end-to-end connectivity from each host? - it is common that a firewall would block some connections but not others depending on legacy rules