Is Kafka a good substitute for syslog-ng?

[danielbb]

With syslog-ng we hit all kinds of limitations from the inability to support TCP, to the inability to write fast enough to disk, and therefore losing vast amounts of UDP data, struggling to have the various F5 LBs to distribute the data evenly to the syslog servers behind the F5s... and all of this led to Kafka as a potential solution. Does anybody use Kafka effectively instead of syslog-ng? By the way, we did look at Splunk Connect for Syslog (SC4S) without much luck.

[Bazsi771]

It's interesting what you write here, syslog-ng was the first one to support TCP based logging, so it definitely supports that.

Dropping of UDP packets is a known issue, but there are a lot of ways to improve that. Here are a couple of blog posts:

https://axoflow.com/syslog-over-udp-kernel-syslog-ng-tuning-avoid-losing-messages/

https://axoflow.com/detect-tcp-udp-message-drops-syslog-telemetry-pipelines/

Kafka is just a message bus, you will need a component to actually receive the messages and then hand them over to kafka. of course syslog-ng can do that:

https://axoflow.com/docs/axosyslog-core/chapter-destinations/configuring-destinations-kafka-c/

[danielbb]

It comes down to whether Kafka can transport UDP and TCP reliably from all the devices to the various Splunk clusters.

[isoutamo]

UDP is not reliable protocol by design! With it you will always lost events. Only question is, how much? Answer is unknown amount.