Getting Data In
Highlighted

How to troubleshoot Syslog-ng -> Splunk issue?

Builder

Hello,

My Splunk installation is configured to ingest data from many different sources. Approximately half of the sources are direct (device -> Splunk) and the other half are indexed from a syslog-ng server (device -> syslog -> Splunk). A few days ago, Splunk stopped indexing all data from the syslog server (about 10 different sourcetypes). I checked connectivity between the syslog server and Splunk and everything seems fine. I also rebooted the syslog server and Splunk. So at this point, I'm not sure what else to do or how to investigate this issue since it's not specific to one source/sourcetype. How can I troubleshoot this issue? Thanks.

Tags (2)
0 Karma
Highlighted

Re: How to troubleshoot Syslog-ng -> Splunk issue?

SplunkTrust
SplunkTrust

Sounds like a problem with Syslog-ng.

  1. Are you receiving events with Syslog-ng?
  2. Are you writing to file, or using tcp/udp to Splunk? ( Syslog -> NG -> file -> Splunk ) OR (Syslog -> NG -> syslog -> Splunk)
  3. Are you using a forwarder at all?
Highlighted

Re: How to troubleshoot Syslog-ng -> Splunk issue?

Builder

Hi,

  1. Yes, syslog-ng is receiving events.
  2. Writing to a file: syslog-ng -> file -> Splunk.
  3. I don't think so but how can I check if I am using a forwarder? Splunk professional services setup our Splunk and I'm still learning the ropes.

Thanks

0 Karma
Highlighted

Re: How to troubleshoot Syslog-ng -> Splunk issue?

SplunkTrust
SplunkTrust

do you have "/opt/splunk" or "/opt/splunkforwarder" on the system?

0 Karma
Highlighted

Re: How to troubleshoot Syslog-ng -> Splunk issue?

Builder

I didn't find /opt/splunk or /opt/splunkforwarder in any of the splunk directories. I do have /opt/ but the directory is empty.

0 Karma
Highlighted

Re: How to troubleshoot Syslog-ng -> Splunk issue?

SplunkTrust
SplunkTrust

What OS do you have?

0 Karma
Highlighted

Re: How to troubleshoot Syslog-ng -> Splunk issue?

Builder

We're running Linux 2.6.32-5...

0 Karma
Highlighted

Re: How to troubleshoot Syslog-ng -> Splunk issue?

Builder

After some digging around, we figured it out: our splunk forwarder service wasn't running on our syslog server. As soon as we started it, we started to see data from our syslog sources in Splunk.

View solution in original post

0 Karma