Archive
Highlighted

Best practices to send multiple devices to a single indexer via syslog

Explorer

We have a deployment with approximately 500 linux systems that are sending logs via syslog on a single indexer. In some cases we notice that some logs are being lost before they arrive.

Is it possible that the indexer have a limit while processing the log flow? . Is there any best practices in this situation?

Thanks in advance.

0 Karma
Highlighted

Re: Best practices to send multiple devices to a single indexer via syslog

SplunkTrust
SplunkTrust

Use a dedicated Syslog server (or servers) to centralize all your syslog traffic, write to text files and then install a universal forwarder locally to read those files.

Some advantages:

  • Offload this from indexer
  • Security (your indexer won't need to listen on restricted ports such as 514)
  • Indexer maintenance or restarts won't affect your syslog traffic
  • Easier to classify, parse, route or even drop your data
  • You could even create a highly-available solution by combining multiple syslog servers and virtual IPs, etc

Couple of links you might find useful:

Hope that helps.

Thanks,
J

View solution in original post

Highlighted

Re: Best practices to send multiple devices to a single indexer via syslog

Explorer

thanks javier for your response. Its help me a lot.

Now, I am thinking to install a heavy forwarder that acts as a syslog server, so I will send all the syslog data to it. Do you know if the heavy forwarder allows load balancing? or if it manages the buffer of events sent to the indexer.

0 Karma
Highlighted

Re: Best practices to send multiple devices to a single indexer via syslog

SplunkTrust
SplunkTrust

I wouldn't install a Heavy Forwarder. See this:

https://www.splunk.com/blog/2016/12/12/universal-or-heavy-that-is-the-question.html

Either install a universal forwarder that listens to whichever Syslog ports you are using or install a dedicated Syslog server (Syslog-NG, rsyslog) that receives your traffic and writes to text files and then a UF in the same server to read those files.

Installing an HF to do this job would be like trying to kill a fly with a bazooka.

0 Karma
Highlighted

Re: Best practices to send multiple devices to a single indexer via syslog

Explorer

Thanks you all very much!

0 Karma
Highlighted

Re: Best practices to send multiple devices to a single indexer via syslog

Ultra Champion

"notice that some logs are being lost before they arrive" - remember that syslog is likely being sent by UDP which is lossy by nature (unlike TCP). So there's a stronger chance that the data is never arriving at your indexer. Also, if sending over UDP and your indexer is restarting, I'm pretty sure all that data is just lost.

0 Karma
Highlighted

Re: Best practices to send multiple devices to a single indexer via syslog

Ultra Champion

@javiergn is spot on. Here's another way of looking at the problem:
You should consider if those linux systems really need to even sent by syslog. A stronger solution would be to have a Universal Forwarder installed directly on those endpoints, directly collecting the data, also able to collect non-syslog data (like OS metrics), send the data over SSL (secure and not lossy like UDP), buffer the data when indexer not around, and scale with an increase in indexers.

0 Karma