All Apps and Add-ons

Splunk Add-on for Kafka: spikes with very long gaps on low throughput

sonicant
Path Finder

I'm using the Splunk Add-on for Kafka to collect events from a kafka topic. The kafka server was on the same machine with splunk, so network should not be an issue.
There is a shell script continuously populating events into a certain kafka topic with 16 partitions (let's say topic "A"), when I use kafka-console-consumer.sh to check topic "A", it prints events continuously which is just as expected. So I thought my kafka setup was correct.

Now the problem is,
When I was populating the events into the topic "A" in normal speed (50-80 events per minute), the splunk add-on for Kafka fetches messages in a strange behavior, it got hundreds of events in 1 minute but "sleep" for another 6-7 minutes. Just like the chart below,
alt text

But, when I increased the populating speed to 1000 msg/s or 5000 msg/s, splunk got those messages immediately and in expected throughput, please see the screenshots below,

alt text

It seems like there is some "buffer" in the add-on which will wait for certain count of messages to flush into splunk...

btw, the Kafka version is:
kafka_2.12-0.10.2.0

the inputs.conf content:

[kafka_mod]
interval = 5

[kafka_mod://kfk1]
index = aaa
kafka_cluster = bbb
kafka_partition_offset = earliest
kafka_topic = A
kafka_topic_group = splunk
kafka_partition = 0

[kafka_mod://kfk2]
index = aaa
kafka_cluster = bbb
kafka_partition_offset = earliest
kafka_topic = A
kafka_topic_group = splunk
kafka_partition = 1

[kafka_mod://kfk3]
index = aaa
kafka_cluster = bbb
kafka_partition_offset = earliest
kafka_topic = A
kafka_topic_group = splunk
kafka_partition = 2

...
//16 partitions in total
...
1 Solution

sonicant
Path Finder

The problem was solved, my colleague found some parameters controlling the batch behavior in the add-on.

View solution in original post

sonicant
Path Finder

The problem was solved, my colleague found some parameters controlling the batch behavior in the add-on.

sivakumarsamy
New Member

How do we get Kafka Key Value pair in Splunk. Always we see only Kafka Value, we need Kafka Key as well

0 Karma

davidepiotti
Explorer

I have the same problem: I have a script writing 1 event per second on a Kafka environment but the Splunk Add-on for Kafka collects more or less 60 events on spikes every one minute.
How did you end up solving the issue?

0 Karma

sonicant
Path Finder

I just tested with kafka_2.10-0.8.2.2, the same result.

0 Karma
Get Updates on the Splunk Community!

Advanced Splunk Data Management Strategies

Join us on Wednesday, May 14, 2025, at 11 AM PDT / 2 PM EDT for an exclusive Tech Talk that delves into ...

Uncovering Multi-Account Fraud with Splunk Banking Analytics

Last month, I met with a Senior Fraud Analyst at a nationally recognized bank to discuss their recent success ...

Secure Your Future: A Deep Dive into the Compliance and Security Enhancements for the ...

What has been announced?  In the blog, “Preparing your Splunk Environment for OpensSSL3,”we announced the ...