Splunk stopped indexing syslog (udp:514) data

[Genti]

@ a customers site:
"
Splunk was indexing udp:514 data until this Wednesday when it stopped altogether. Using TCPDump i was able to confirm that data is coming into the server but splunk is not indexing it.

The splunk service is started as root and no other data is missing.
"

[Genti · Accepted solution]

checking the splunkd.logs and metrics.logs showed no sign of Blocked data. There were also no other errors related to UDP:514 within the splunkd.logs

On the other hand, metrics.logs show that splunk was listening and indexing data till a certain point in time, and afterwords no other data was coming in. This does not match the statement from the customer...

One thing to keep in mind though is that tcpDump listens to the connection socket prior to any application, so there might be some firewall or an app that is blocking it from going into splunk.

Try a netstat -an | grep 514 to check if there is another app listening to the port. In the customers case it ended up being syslog starting up on wednesday and blocking the data from going to splunk...