Splunk Add-on for Kafka: Why is there low performance after configuration?

[wangjiaji]

I'm using Splunk Add-on for Kafka to collect data from a Kafka cluster, I believe the configs are OK because data is coming, but the transmission speed is very low, about 200 msg/min (the transmission speed into Kafka is about 1000 msg/min), but once in an hour or two, there are 20k msg/min for 1-2 minutes. The Kafka cluster is on the same machine, so I think network won't be a problem. I have repeated the same configuration on 3 different machines, and the same thing happens, what could've caused this issue?

[DalJeanis]

This page refers to batch mode... https://www.splunk.com/blog/2015/10/12/achieving-scale-with-the-kafka-modular-input/

Enable Batch Mode. This will buffer events in memory until the batch buffer is flushed depending on how you tune the flush settings.You can tune the size of the batch buffer depending on the scale of the throughput in your Kafka environment ie: higher throughput => larger batch buffer will be more optimal.

Sounds to me like you want a smaller batch size.