We have a deployment with approximately 500 linux systems that are sending logs via syslog on a single indexer. In some cases we notice that some logs are being lost before they arrive.
Is it possible that the indexer have a limit while processing the log flow? . Is there any best practices in this situation?
Thanks in advance.
Use a dedicated Syslog server (or servers) to centralize all your syslog traffic, write to text files and then install a universal forwarder locally to read those files.
Couple of links you might find useful:
Hope that helps.
thanks javier for your response. Its help me a lot.
Now, I am thinking to install a heavy forwarder that acts as a syslog server, so I will send all the syslog data to it. Do you know if the heavy forwarder allows load balancing? or if it manages the buffer of events sent to the indexer.
I wouldn't install a Heavy Forwarder. See this:
Either install a universal forwarder that listens to whichever Syslog ports you are using or install a dedicated Syslog server (Syslog-NG, rsyslog) that receives your traffic and writes to text files and then a UF in the same server to read those files.
Installing an HF to do this job would be like trying to kill a fly with a bazooka.
"notice that some logs are being lost before they arrive" - remember that syslog is likely being sent by UDP which is lossy by nature (unlike TCP). So there's a stronger chance that the data is never arriving at your indexer. Also, if sending over UDP and your indexer is restarting, I'm pretty sure all that data is just lost.
@javiergn is spot on. Here's another way of looking at the problem:
You should consider if those linux systems really need to even sent by syslog. A stronger solution would be to have a Universal Forwarder installed directly on those endpoints, directly collecting the data, also able to collect non-syslog data (like OS metrics), send the data over SSL (secure and not lossy like UDP), buffer the data when indexer not around, and scale with an increase in indexers.