I'm trying to read and index messages that come from a Juniper Pulse device using syslog protocol. I used the "Data Input" menu and add 10520/UDP as input port and bind it to a new index.
When I listen to port using tcpdump, I can see the messages from console, however, Splunk can't see and index the incoming data. I tried different sourcetypes like syslog, __singleline etc...
When I run
netstat -tunalp | grep 10520, I could see that Splunk is listening on udp port 10520.
How can I debug this situation? What's your advice?
Start with http://docs.splunk.com/Documentation/Splunk/6.4.1/Troubleshooting/Cantfinddata and https://answers.splunk.com/answers/221885/how-to-troubleshoot-why-i-can-see-network-traffic.html just in case something there helps.
I would try a combination of the splunkd logs and using strace on the Splunk process. Also, enable debug and sifting through the results may be useful.
My recommendation is to take a sample of the data and put it into a file on your local machine. Then go to add data in the Splunk GUI and upload from your local machine. You will then be brought to a screen where it tries to determine a sourcetype. You can play around with different sourcetype settings. When you try one like syslog for example make sure that linebreaking is happening as you'd expect and the a timestamp is extracted from the data.
The other thing to check would be to look at the splunkd.log in index=_internal to check for errors. That could give you a more specific idea of what might be wrong.