Getting Data In

How to troubleshoot Syslog-ng -> Splunk issue?

echojacques
Builder

Hello,

My Splunk installation is configured to ingest data from many different sources. Approximately half of the sources are direct (device -> Splunk) and the other half are indexed from a syslog-ng server (device -> syslog -> Splunk). A few days ago, Splunk stopped indexing all data from the syslog server (about 10 different sourcetypes). I checked connectivity between the syslog server and Splunk and everything seems fine. I also rebooted the syslog server and Splunk. So at this point, I'm not sure what else to do or how to investigate this issue since it's not specific to one source/sourcetype. How can I troubleshoot this issue? Thanks.

Tags (2)
0 Karma
1 Solution

echojacques
Builder

After some digging around, we figured it out: our splunk forwarder service wasn't running on our syslog server. As soon as we started it, we started to see data from our syslog sources in Splunk.

View solution in original post

0 Karma

echojacques
Builder

After some digging around, we figured it out: our splunk forwarder service wasn't running on our syslog server. As soon as we started it, we started to see data from our syslog sources in Splunk.

0 Karma

alacercogitatus
SplunkTrust
SplunkTrust

Sounds like a problem with Syslog-ng.

  1. Are you receiving events with Syslog-ng?
  2. Are you writing to file, or using tcp/udp to Splunk? ( Syslog -> NG -> file -> Splunk ) OR (Syslog -> NG -> syslog -> Splunk)
  3. Are you using a forwarder at all?

echojacques
Builder

We're running Linux 2.6.32-5...

0 Karma

alacercogitatus
SplunkTrust
SplunkTrust

What OS do you have?

0 Karma

echojacques
Builder

I didn't find /opt/splunk or /opt/splunkforwarder in any of the splunk directories. I do have /opt/ but the directory is empty.

0 Karma

alacercogitatus
SplunkTrust
SplunkTrust

do you have "/opt/splunk" or "/opt/splunkforwarder" on the system?

0 Karma

echojacques
Builder

Hi,

  1. Yes, syslog-ng is receiving events.
  2. Writing to a file: syslog-ng -> file -> Splunk.
  3. I don't think so but how can I check if I am using a forwarder? Splunk professional services setup our Splunk and I'm still learning the ropes.

Thanks

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...