Splunk Add-on for Kafka: spikes with very long gaps on low throughput

[sonicant]

I'm using the Splunk Add-on for Kafka to collect events from a kafka topic. The kafka server was on the same machine with splunk, so network should not be an issue.
There is a shell script continuously populating events into a certain kafka topic with 16 partitions (let's say topic "A"), when I use kafka-console-consumer.sh to check topic "A", it prints events continuously which is just as expected. So I thought my kafka setup was correct.

Now the problem is,
When I was populating the events into the topic "A" in normal speed (50-80 events per minute), the splunk add-on for Kafka fetches messages in a strange behavior, it got hundreds of events in 1 minute but "sleep" for another 6-7 minutes. Just like the chart below,

But, when I increased the populating speed to 1000 msg/s or 5000 msg/s, splunk got those messages immediately and in expected throughput, please see the screenshots below,

It seems like there is some "buffer" in the add-on which will wait for certain count of messages to flush into splunk...

btw, the Kafka version is:
kafka_2.12-0.10.2.0

the inputs.conf content:

[kafka_mod]
interval = 5

[kafka_mod://kfk1]
index = aaa
kafka_cluster = bbb
kafka_partition_offset = earliest
kafka_topic = A
kafka_topic_group = splunk
kafka_partition = 0

[kafka_mod://kfk2]
index = aaa
kafka_cluster = bbb
kafka_partition_offset = earliest
kafka_topic = A
kafka_topic_group = splunk
kafka_partition = 1

[kafka_mod://kfk3]
index = aaa
kafka_cluster = bbb
kafka_partition_offset = earliest
kafka_topic = A
kafka_topic_group = splunk
kafka_partition = 2

...
//16 partitions in total
...

[sonicant · Accepted solution]

The problem was solved, my colleague found some parameters controlling the batch behavior in the add-on.

[sivakumarsamy]

How do we get Kafka Key Value pair in Splunk. Always we see only Kafka Value, we need Kafka Key as well

[davidepiotti]

I have the same problem: I have a script writing 1 event per second on a Kafka environment but the Splunk Add-on for Kafka collects more or less 60 events on spikes every one minute.
How did you end up solving the issue?

[sonicant]

I just tested with kafka_2.10-0.8.2.2, the same result.