How to troubleshoot why Splunk isn't reading and indexing incoming syslog messages?

[ozirus]

Hi,

I'm trying to read and index messages that come from a Juniper Pulse device using syslog protocol. I used the "Data Input" menu and add 10520/UDP as input port and bind it to a new index.

When I listen to port using tcpdump, I can see the messages from console, however, Splunk can't see and index the incoming data. I tried different sourcetypes like syslog, __singleline etc...

When I run netstat -tunalp | grep 10520, I could see that Splunk is listening on udp port 10520.

How can I debug this situation? What's your advice?

[diogofgm]

have you tried to use TCP input instead?

[ozirus]

It also doesn't work

[masonmorales]

Does the data show up in the index if you search All Time?

[ozirus]

No. There is no data in any way.

[jtrucks]

Start with http://docs.splunk.com/Documentation/Splunk/6.4.1/Troubleshooting/Cantfinddata and https://answers.splunk.com/answers/221885/how-to-troubleshoot-why-i-can-see-network-traffic.html just in case something there helps.

I would try a combination of the splunkd logs and using strace on the Splunk process. Also, enable debug and sifting through the results may be useful.

[craigv_splunk]

My recommendation is to take a sample of the data and put it into a file on your local machine. Then go to add data in the Splunk GUI and upload from your local machine. You will then be brought to a screen where it tries to determine a sourcetype. You can play around with different sourcetype settings. When you try one like syslog for example make sure that linebreaking is happening as you'd expect and the a timestamp is extracted from the data.

The other thing to check would be to look at the splunkd.log in index=_internal to check for errors. That could give you a more specific idea of what might be wrong.

[ozirus]

How can I achive this data-import for syslog? tcpdump gives messy ASCII data when I listen syslog port. Any suggestion?