All Apps and Add-ons

Kafka Messaging Modular Input: Kafka consumer is apparently connected, but how do we troubleshoot why we see no data?

Path Finder

We have followed the troubleshooting steps, but are still not able to get this input working:

1) JAVA_HOME is set and java is in the path (for openJDK)
2) Splunk 6.2.3
3) Java OpenJDK, 1.7.0_85
4) Kafka version 0.8.1.1.
5) on linux
6) the only errors in splunkd.log are the SLF4J errors shown below
7) Running the command line invocation for the scheme doesn't show any errors, just prints out what the arguments are and their descriptions.

Additional troubleshooting performed:
On both the splunk forwarder host and the host running Kafka, we used netstat -anp to verify that that the Java process on splunk was connected to kafka, and we could see an ESTABLISHED socket.

Otherwise, all we see in splunkd.log is shown below. Is there a way to enable DEBUG or further troubleshooting information for the Java process/kafka consumer?

Log messages:

08-28-2015 17:26:12.618 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/kafka_ta/bin/kafka.py" SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
08-28-2015 17:26:12.618 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/kafka_ta/bin/kafka.py" SLF4J: Defaulting to no-operation (NOP) logger implementation
08-28-2015 17:26:12.618 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/kafka_ta/bin/kafka.py" SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
08-28-2015 17:26:11.298 +0000 INFO  ExecProcessor -     interval: run once
08-28-2015 17:26:11.298 +0000 INFO  ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/kafka_ta/bin/kafka.py

Path Finder

@jnicholsenernoc did you ever resolve this issue? We are facing a similar thing... https://answers.splunk.com/answers/323066/kafka-messaging-modular-input-messages-are-consume.html

0 Karma

Contributor

When we setup the kafka_ta, we installed it on the indexers and on the search heads. Then, from the UI, we added an input and put our brokers into the zookeeper hosts, gave it a group id, and put in the topic name. We never had to involve the forwarders. This sound crazy?

Path Finder

No, not crazy. We did however try to install it directly on a search head and that didn't seem to work either.

0 Karma

Ultra Champion

In a distributed Splunk architecture you should only install the kafka modular input on a forwarder or indexer. Not on a search head.
You then configure your stanza by manually editing the inputs.conf file.

0 Karma

Contributor

@jnicholsenernoc - When we setup the kafka_ta, we installed it on the indexers and on the search heads. Then, from the UI, we added an input and put our brokers into the zookeeper hosts, gave it a group id, and put in the topic name. We never had to involve the forwarders. This sound crazy?

0 Karma

Path Finder

Any ideas on how to troubleshoot this further? I tried the setup from the start with new installation, still seems to be connected but not actually reading data.

0 Karma

Ultra Champion

how are you searching in Splunk ? is your time span correct for example ?

Can you see the messages on the wire ? ie: using wireshark

0 Karma

Path Finder

For the search in splunk, it is filtering by the sourcetype=kafka and search all time for it in the index (currently sending it to main).

We haven't tried to wireshark it to see if it is passing any data. We can see that it is connected at the network socket level but can try to do this.

Is there a way to set the JAVA process to log more DEBUG information?

0 Karma

Ultra Champion

Can you post your full inputs.conf stanza for your kafka setup ?

0 Karma

Path Finder

[kafka://logging_spark_bdap_dev]
group_id = splunk_dev
index = main
sourcetype = kafka
topic_name = logging_spark_bdap_dev
zookeeper_connect_chroot = kafka-development
zookeeper_connect_host = zookeeper-1.x.x.x
zookeeper_connect_port = 2181
zookeeper_session_timeout_ms = 5000

0 Karma