My Splunk installation is configured to ingest data from many different sources. Approximately half of the sources are direct (device -> Splunk) and the other half are indexed from a syslog-ng server (device -> syslog -> Splunk). A few days ago, Splunk stopped indexing all data from the syslog server (about 10 different sourcetypes). I checked connectivity between the syslog server and Splunk and everything seems fine. I also rebooted the syslog server and Splunk. So at this point, I'm not sure what else to do or how to investigate this issue since it's not specific to one source/sourcetype. How can I troubleshoot this issue? Thanks.
Sounds like a problem with Syslog-ng.
I didn't find /opt/splunk or /opt/splunkforwarder in any of the splunk directories. I do have /opt/ but the directory is empty.
After some digging around, we figured it out: our splunk forwarder service wasn't running on our syslog server. As soon as we started it, we started to see data from our syslog sources in Splunk.