topic Re: Kafka Messaging Modular Input: Kafka consumer is apparently connected, but how do we troubleshoot why we see no data? in All Apps and Add-ons

Kafka Messaging Modular Input: Kafka consumer is apparently connected, but how do we troubleshoot why we see no data?

jnicholsenernoc — Tue, 29 Sep 2020 07:08:36 GMT

We have followed the troubleshooting steps, but are still not able to get this input working:

1) JAVA_HOME is set and java is in the path (for openJDK)
2) Splunk 6.2.3
3) Java OpenJDK, 1.7.0_85
4) Kafka version 0.8.1.1.
5) on linux
6) the only errors in splunkd.log are the SLF4J errors shown below
7) Running the command line invocation for the scheme doesn't show any errors, just prints out what the arguments are and their descriptions.

Additional troubleshooting performed:
On both the splunk forwarder host and the host running Kafka, we used netstat -anp to verify that that the Java process on splunk was connected to kafka, and we could see an ESTABLISHED socket.

Otherwise, all we see in splunkd.log is shown below. Is there a way to enable DEBUG or further troubleshooting information for the Java process/kafka consumer?

Log messages:

08-28-2015 17:26:12.618 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/kafka_ta/bin/kafka.py" SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
08-28-2015 17:26:12.618 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/kafka_ta/bin/kafka.py" SLF4J: Defaulting to no-operation (NOP) logger implementation
08-28-2015 17:26:12.618 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/kafka_ta/bin/kafka.py" SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
08-28-2015 17:26:11.298 +0000 INFO  ExecProcessor -     interval: run once
08-28-2015 17:26:11.298 +0000 INFO  ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/kafka_ta/bin/kafka.py

Re: Kafka Messaging Modular Input: Kafka consumer is apparently connected, but how do we troubleshoot why we see no data?

Damien_Dallimor — Sat, 29 Aug 2015 11:12:34 GMT

Can you post your full inputs.conf stanza for your kafka setup ?

Re: Kafka Messaging Modular Input: Kafka consumer is apparently connected, but how do we troubleshoot why we see no data?

jnicholsenernoc — Tue, 29 Sep 2020 07:08:50 GMT

[kafka://logging_spark_bdap_dev]
group_id = splunk_dev
index = main
sourcetype = kafka
topic_name = logging_spark_bdap_dev
zookeeper_connect_chroot = kafka-development
zookeeper_connect_host = zookeeper-1.x.x.x
zookeeper_connect_port = 2181
zookeeper_session_timeout_ms = 5000

Re: Kafka Messaging Modular Input: Kafka consumer is apparently connected, but how do we troubleshoot why we see no data?

jnicholsenernoc — Wed, 09 Sep 2015 13:56:07 GMT

Any ideas on how to troubleshoot this further? I tried the setup from the start with new installation, still seems to be connected but not actually reading data.

Re: Kafka Messaging Modular Input: Kafka consumer is apparently connected, but how do we troubleshoot why we see no data?

Damien_Dallimor — Thu, 10 Sep 2015 10:32:18 GMT

how are you searching in Splunk ? is your time span correct for example ?

Can you see the messages on the wire ? ie: using wireshark

Re: Kafka Messaging Modular Input: Kafka consumer is apparently connected, but how do we troubleshoot why we see no data?

jaredlaney — Thu, 10 Sep 2015 12:40:31 GMT

@jnicholsenernoc - When we setup the kafka_ta, we installed it on the indexers and on the search heads. Then, from the UI, we added an input and put our brokers into the zookeeper hosts, gave it a group id, and put in the topic name. We never had to involve the forwarders. This sound crazy?

Re: Kafka Messaging Modular Input: Kafka consumer is apparently connected, but how do we troubleshoot why we see no data?

jaredlaney — Thu, 10 Sep 2015 12:41:33 GMT

When we setup the kafka_ta, we installed it on the indexers and on the search heads. Then, from the UI, we added an input and put our brokers into the zookeeper hosts, gave it a group id, and put in the topic name. We never had to involve the forwarders. This sound crazy?

Re: Kafka Messaging Modular Input: Kafka consumer is apparently connected, but how do we troubleshoot why we see no data?

jnicholsenernoc — Thu, 10 Sep 2015 13:16:26 GMT

For the search in splunk, it is filtering by the sourcetype=kafka and search all time for it in the index (currently sending it to main).

We haven't tried to wireshark it to see if it is passing any data. We can see that it is connected at the network socket level but can try to do this.

Is there a way to set the JAVA process to log more DEBUG information?

Re: Kafka Messaging Modular Input: Kafka consumer is apparently connected, but how do we troubleshoot why we see no data?

jnicholsenernoc — Thu, 10 Sep 2015 13:17:11 GMT

No, not crazy. We did however try to install it directly on a search head and that didn't seem to work either.

Re: Kafka Messaging Modular Input: Kafka consumer is apparently connected, but how do we troubleshoot why we see no data?

Damien_Dallimor — Thu, 10 Sep 2015 13:34:19 GMT

In a distributed Splunk architecture you should only install the kafka modular input on a forwarder or indexer. Not on a search head.
You then configure your stanza by manually editing the inputs.conf file.

Re: Kafka Messaging Modular Input: Kafka consumer is apparently connected, but how do we troubleshoot why we see no data?

petehmrc — Tue, 03 Nov 2015 09:14:21 GMT

@jnicholsenernoc did you ever resolve this issue? We are facing a similar thing... https://answers.splunk.com/answers/323066/kafka-messaging-modular-input-messages-are-consume.html