Getting Data In

Splunk stopped indexing syslog (udp:514) data

Genti
Splunk Employee
Splunk Employee

@ a customers site:
"
Splunk was indexing udp:514 data until this Wednesday when it stopped altogether. Using TCPDump i was able to confirm that data is coming into the server but splunk is not indexing it.

The splunk service is started as root and no other data is missing.
"

Tags (3)
0 Karma
1 Solution

Genti
Splunk Employee
Splunk Employee

checking the splunkd.logs and metrics.logs showed no sign of Blocked data. There were also no other errors related to UDP:514 within the splunkd.logs

On the other hand, metrics.logs show that splunk was listening and indexing data till a certain point in time, and afterwords no other data was coming in. This does not match the statement from the customer...

One thing to keep in mind though is that tcpDump listens to the connection socket prior to any application, so there might be some firewall or an app that is blocking it from going into splunk.

Try a netstat -an | grep 514 to check if there is another app listening to the port. In the customers case it ended up being syslog starting up on wednesday and blocking the data from going to splunk...

View solution in original post

Genti
Splunk Employee
Splunk Employee

checking the splunkd.logs and metrics.logs showed no sign of Blocked data. There were also no other errors related to UDP:514 within the splunkd.logs

On the other hand, metrics.logs show that splunk was listening and indexing data till a certain point in time, and afterwords no other data was coming in. This does not match the statement from the customer...

One thing to keep in mind though is that tcpDump listens to the connection socket prior to any application, so there might be some firewall or an app that is blocking it from going into splunk.

Try a netstat -an | grep 514 to check if there is another app listening to the port. In the customers case it ended up being syslog starting up on wednesday and blocking the data from going to splunk...

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...