topic Re: How to troubleshoot Syslog-ng -> Splunk issue? in Getting Data In

How to troubleshoot Syslog-ng -> Splunk issue?

echojacques — Thu, 05 Dec 2013 15:51:02 GMT

Hello,

My Splunk installation is configured to ingest data from many different sources. Approximately half of the sources are direct (device -> Splunk) and the other half are indexed from a syslog-ng server (device -> syslog -> Splunk). A few days ago, Splunk stopped indexing all data from the syslog server (about 10 different sourcetypes). I checked connectivity between the syslog server and Splunk and everything seems fine. I also rebooted the syslog server and Splunk. So at this point, I'm not sure what else to do or how to investigate this issue since it's not specific to one source/sourcetype. How can I troubleshoot this issue? Thanks.

Re: How to troubleshoot Syslog-ng -> Splunk issue?

alacercogitatus — Thu, 05 Dec 2013 16:25:44 GMT

Sounds like a problem with Syslog-ng.

Are you receiving events with Syslog-ng?
Are you writing to file, or using tcp/udp to Splunk? ( Syslog -> NG -> file -> Splunk ) OR (Syslog -> NG -> syslog -> Splunk)
Are you using a forwarder at all?

Re: How to troubleshoot Syslog-ng -> Splunk issue?

echojacques — Thu, 05 Dec 2013 17:03:49 GMT

Hi,

Yes, syslog-ng is receiving events.
Writing to a file: syslog-ng -> file -> Splunk.
I don't think so but how can I check if I am using a forwarder? Splunk professional services setup our Splunk and I'm still learning the ropes.

Thanks

Re: How to troubleshoot Syslog-ng -> Splunk issue?

alacercogitatus — Thu, 05 Dec 2013 17:11:31 GMT

do you have "/opt/splunk" or "/opt/splunkforwarder" on the system?

Re: How to troubleshoot Syslog-ng -> Splunk issue?

echojacques — Thu, 05 Dec 2013 17:25:28 GMT

I didn't find /opt/splunk or /opt/splunkforwarder in any of the splunk directories. I do have /opt/ but the directory is empty.

Re: How to troubleshoot Syslog-ng -> Splunk issue?

alacercogitatus — Thu, 05 Dec 2013 17:38:37 GMT

What OS do you have?

Re: How to troubleshoot Syslog-ng -> Splunk issue?

echojacques — Thu, 05 Dec 2013 18:04:24 GMT

We're running Linux 2.6.32-5...

Re: How to troubleshoot Syslog-ng -> Splunk issue?

echojacques — Thu, 05 Dec 2013 23:05:27 GMT

After some digging around, we figured it out: our splunk forwarder service wasn't running on our syslog server. As soon as we started it, we started to see data from our syslog sources in Splunk.