How to troubleshoot Syslog-ng -> Splunk issue?

[echojacques]

Hello,

My Splunk installation is configured to ingest data from many different sources. Approximately half of the sources are direct (device -> Splunk) and the other half are indexed from a syslog-ng server (device -> syslog -> Splunk). A few days ago, Splunk stopped indexing all data from the syslog server (about 10 different sourcetypes). I checked connectivity between the syslog server and Splunk and everything seems fine. I also rebooted the syslog server and Splunk. So at this point, I'm not sure what else to do or how to investigate this issue since it's not specific to one source/sourcetype. How can I troubleshoot this issue? Thanks.

[echojacques · Accepted solution]

After some digging around, we figured it out: our splunk forwarder service wasn't running on our syslog server. As soon as we started it, we started to see data from our syslog sources in Splunk.

[alacercogitatus]

Sounds like a problem with Syslog-ng.

1. Are you receiving events with Syslog-ng?
2. Are you writing to file, or using tcp/udp to Splunk? ( Syslog -> NG -> file -> Splunk ) OR (Syslog -> NG -> syslog -> Splunk)
3. Are you using a forwarder at all?

[echojacques]

We're running Linux 2.6.32-5...

[alacercogitatus]

What OS do you have?

[echojacques]

I didn't find /opt/splunk or /opt/splunkforwarder in any of the splunk directories. I do have /opt/ but the directory is empty.

[alacercogitatus]

do you have "/opt/splunk" or "/opt/splunkforwarder" on the system?

[echojacques]

Hi,

1. Yes, syslog-ng is receiving events.
2. Writing to a file: syslog-ng -> file -> Splunk.
3. I don't think so but how can I check if I am using a forwarder? Splunk professional services setup our Splunk and I'm still learning the ropes.

Thanks