Installation

Kafka Splunk Sink connector: All channels have back pressure

rpollard001
New Member

Hi,

I have a Kafka cluster running, and periodically, the active controller fails. This causes issues with the Splunk sink connector and therefore stops the process of streaming audit data from Cloudera to Splunk.

The error I'm getting over and over is:

[2019-02-21 14:54:10,672] INFO add 1 failed batches (com.splunk.kafka.connect.SplunkSinkTask:322)
[2019-02-21 14:54:10,672] INFO total failed batches 263 (com.splunk.kafka.connect.SplunkSinkTask:47)
[2019-02-21 14:54:10,672] ERROR failed to send batch (com.splunk.kafka.connect.SplunkSinkTask:261)
com.splunk.hecclient.HecException: All channels have back pressure
    at com.splunk.hecclient.LoadBalancer.send(LoadBalancer.java:62)
    at com.splunk.hecclient.Hec.send(Hec.java:233)
    at com.splunk.kafka.connect.SplunkSinkTask.send(SplunkSinkTask.java:257)
    at com.splunk.kafka.connect.SplunkSinkTask.handleFailedBatches(SplunkSinkTask.java:127)
    at com.splunk.kafka.connect.SplunkSinkTask.put(SplunkSinkTask.java:62)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:495)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:288)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:198)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:166)
    at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:170)
    at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:214)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

I have restarted and reset the sink connector configuration. It is still spitting out this error after an hour.

NOTE: This setup has been working for almost 2 weeks. It has only started doing this today. It did do this about 5 days ago. It came up with the same error and then corrected itself.

Thanks!

Tags (2)
0 Karma

rpollard001
New Member

ANSWER:
The problem was the Splunk server was down. Had not received a notice that is was going to be worked on and I assumed the problem was on my side.

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...