syslog-ng in ubuntu suddenly stopped sending logs toward splunk

[ornaldo]

Dear community,

Until yesterday syslog-ng in ubuntu suddenly stopped sending logs toward splunk. 

I have restarted the syslog-ng services, splunk, and splunkforwarder service but still nothing.

Any idea for troubleshooting ?

Thank You

[meetmshah]

Glad it is resolved now!

Feel free to accept answer if the troubleshooting steps helped : )

[meetmshah]

Hello @ornaldo, Can you please confirm - 

1. Are you directly monitoring the port on which syslog-ng is sending data? If so, have you validated through tcpdump that events are actually being ingested on host?
2. If you are performing file monitoring, have you checked if the necessary files are being generated on the host? If not, can you please check syslog-ng configuration?
3. If the ingestion is through file monitoring, can you please check if the files / folders are matching the inputs.conf configuration. For example inputs.conf may monitor abc.* but file name is abd.txt
4. Check the file permission and confirm Splunk has read access to those files
5. Double-check if all the logs are missing from that host? If so, can you verify the connectivity, certificate configurations, queues etc.

[ornaldo · Accepted solution]

Hi there,

It was a problem with syslog-ng.conf.

Also: Error connecting control socket, socket='/var/lib/syslog-ng/syslog-ng.ctl', error='Permission denied'

Another daemon was binding to port 514. We found using netstat> https://community.spiceworks.com/topic/2323445-syslog-ng-stop-sending-logs-to-splunk

We fixed.

Thank You