Splunk Enterprise

Is Kafka a good substitute for syslog-ng?

danielbb
Motivator

With syslog-ng we hit all kinds of limitations from the inability to support TCP, to the inability to write fast enough to disk, and therefore losing vast amounts of UDP data, struggling to have the various F5 LBs to distribute the data evenly to the syslog servers behind the F5s... and all of this led to Kafka as a potential solution. Does anybody use Kafka effectively instead of syslog-ng? By the way, we did look at Splunk Connect for Syslog (SC4S) without much luck.

Labels (1)
0 Karma

Bazsi771
New Member

It's interesting what you write here, syslog-ng was the first one to support TCP based logging, so it definitely supports that.

Dropping of UDP packets is a known issue, but there are a lot of ways to improve that. Here are a couple of blog posts:

https://axoflow.com/syslog-over-udp-kernel-syslog-ng-tuning-avoid-losing-messages/

https://axoflow.com/detect-tcp-udp-message-drops-syslog-telemetry-pipelines/

Kafka is just a message bus, you will need a component to actually receive the messages and then hand them over to kafka. of course syslog-ng can do that:

https://axoflow.com/docs/axosyslog-core/chapter-destinations/configuring-destinations-kafka-c/

 

0 Karma

danielbb
Motivator

It comes down to whether Kafka can transport UDP and TCP reliably from all the devices to the various Splunk clusters.

0 Karma

isoutamo
SplunkTrust
SplunkTrust
UDP is not reliable protocol by design! With it you will always lost events. Only question is, how much? Answer is unknown amount.
Get Updates on the Splunk Community!

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Get More Out of Your Security Practice With a SIEM

Get More Out of Your Security Practice With a SIEMWednesday, July 31, 2024  |  11AM PT / 2PM ETREGISTER ...