Best practices to send multiple devices to a single indexer via syslog

[lightech1]

We have a deployment with approximately 500 linux systems that are sending logs via syslog on a single indexer. In some cases we notice that some logs are being lost before they arrive.

Is it possible that the indexer have a limit while processing the log flow? . Is there any best practices in this situation?

Thanks in advance.

[sloshburch]

@javiergn is spot on. Here's another way of looking at the problem:
You should consider if those linux systems really need to even sent by syslog. A stronger solution would be to have a Universal Forwarder installed directly on those endpoints, directly collecting the data, also able to collect non-syslog data (like OS metrics), send the data over SSL (secure and not lossy like UDP), buffer the data when indexer not around, and scale with an increase in indexers.

[sloshburch]

"notice that some logs are being lost before they arrive" - remember that syslog is likely being sent by UDP which is lossy by nature (unlike TCP). So there's a stronger chance that the data is never arriving at your indexer. Also, if sending over UDP and your indexer is restarting, I'm pretty sure all that data is just lost.

[javiergn · Accepted solution]

Use a dedicated Syslog server (or servers) to centralize all your syslog traffic, write to text files and then install a universal forwarder locally to read those files.

Some advantages:

• Offload this from indexer
• Security (your indexer won't need to listen on restricted ports such as 514)
• Indexer maintenance or restarts won't affect your syslog traffic
• Easier to classify, parse, route or even drop your data
• You could even create a highly-available solution by combining multiple syslog servers and virtual IPs, etc

Couple of links you might find useful:

• https://www.splunk.com/blog/2016/03/11/using-syslog-ng-with-splunk.html
• https://www.splunk.com/blog/2016/05/05/high-performance-syslogging-for-splunk-using-syslog-ng-part-1...
• https://www.splunk.com/blog/2016/05/05/high-performance-syslogging-for-splunk-using-syslog-ng-part-2...
• http://www.georgestarcher.com/splunk-success-with-syslog/

Hope that helps.

Thanks,
J

[lightech1]

Thanks you all very much!

[lightech1]

thanks javier for your response. Its help me a lot.

Now, I am thinking to install a heavy forwarder that acts as a syslog server, so I will send all the syslog data to it. Do you know if the heavy forwarder allows load balancing? or if it manages the buffer of events sent to the indexer.

[javiergn]

I wouldn't install a Heavy Forwarder. See this:

https://www.splunk.com/blog/2016/12/12/universal-or-heavy-that-is-the-question.html

Either install a universal forwarder that listens to whichever Syslog ports you are using or install a dedicated Syslog server (Syslog-NG, rsyslog) that receives your traffic and writes to text files and then a UF in the same server to read those files.

Installing an HF to do this job would be like trying to kill a fly with a bazooka.