All Apps and Add-ons

Splunk Add-on for Kafka: Why is there low performance after configuration?

wangjiaji
Engager

I'm using Splunk Add-on for Kafka to collect data from a Kafka cluster, I believe the configs are OK because data is coming, but the transmission speed is very low, about 200 msg/min (the transmission speed into Kafka is about 1000 msg/min), but once in an hour or two, there are 20k msg/min for 1-2 minutes. The Kafka cluster is on the same machine, so I think network won't be a problem. I have repeated the same configuration on 3 different machines, and the same thing happens, what could've caused this issue?

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

This page refers to batch mode... https://www.splunk.com/blog/2015/10/12/achieving-scale-with-the-kafka-modular-input/

Enable Batch Mode. This will buffer events in memory until the batch buffer is flushed depending on how you tune the flush settings.You can tune the size of the batch buffer depending on the scale of the throughput in your Kafka environment ie: higher throughput => larger batch buffer will be more optimal.

Sounds to me like you want a smaller batch size.

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...