All Apps and Add-ons

Kafka Messaging Modular Input: Kafka consumer is apparently connected, but how do we troubleshoot why we see no data?

jnicholsenernoc
Path Finder

We have followed the troubleshooting steps, but are still not able to get this input working:

1) JAVA_HOME is set and java is in the path (for openJDK)
2) Splunk 6.2.3
3) Java OpenJDK, 1.7.0_85
4) Kafka version 0.8.1.1.
5) on linux
6) the only errors in splunkd.log are the SLF4J errors shown below
7) Running the command line invocation for the scheme doesn't show any errors, just prints out what the arguments are and their descriptions.

Additional troubleshooting performed:
On both the splunk forwarder host and the host running Kafka, we used netstat -anp to verify that that the Java process on splunk was connected to kafka, and we could see an ESTABLISHED socket.

Otherwise, all we see in splunkd.log is shown below. Is there a way to enable DEBUG or further troubleshooting information for the Java process/kafka consumer?

Log messages:

08-28-2015 17:26:12.618 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/kafka_ta/bin/kafka.py" SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
08-28-2015 17:26:12.618 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/kafka_ta/bin/kafka.py" SLF4J: Defaulting to no-operation (NOP) logger implementation
08-28-2015 17:26:12.618 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/kafka_ta/bin/kafka.py" SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
08-28-2015 17:26:11.298 +0000 INFO  ExecProcessor -     interval: run once
08-28-2015 17:26:11.298 +0000 INFO  ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/kafka_ta/bin/kafka.py

petehmrc
Path Finder

@jnicholsenernoc did you ever resolve this issue? We are facing a similar thing... https://answers.splunk.com/answers/323066/kafka-messaging-modular-input-messages-are-consume.html

0 Karma

jaredlaney
Contributor

When we setup the kafka_ta, we installed it on the indexers and on the search heads. Then, from the UI, we added an input and put our brokers into the zookeeper hosts, gave it a group id, and put in the topic name. We never had to involve the forwarders. This sound crazy?

jnicholsenernoc
Path Finder

No, not crazy. We did however try to install it directly on a search head and that didn't seem to work either.

0 Karma

Damien_Dallimor
Ultra Champion

In a distributed Splunk architecture you should only install the kafka modular input on a forwarder or indexer. Not on a search head.
You then configure your stanza by manually editing the inputs.conf file.

0 Karma

jaredlaney
Contributor

@jnicholsenernoc - When we setup the kafka_ta, we installed it on the indexers and on the search heads. Then, from the UI, we added an input and put our brokers into the zookeeper hosts, gave it a group id, and put in the topic name. We never had to involve the forwarders. This sound crazy?

0 Karma

jnicholsenernoc
Path Finder

Any ideas on how to troubleshoot this further? I tried the setup from the start with new installation, still seems to be connected but not actually reading data.

0 Karma

Damien_Dallimor
Ultra Champion

how are you searching in Splunk ? is your time span correct for example ?

Can you see the messages on the wire ? ie: using wireshark

0 Karma

jnicholsenernoc
Path Finder

For the search in splunk, it is filtering by the sourcetype=kafka and search all time for it in the index (currently sending it to main).

We haven't tried to wireshark it to see if it is passing any data. We can see that it is connected at the network socket level but can try to do this.

Is there a way to set the JAVA process to log more DEBUG information?

0 Karma

Damien_Dallimor
Ultra Champion

Can you post your full inputs.conf stanza for your kafka setup ?

0 Karma

jnicholsenernoc
Path Finder

[kafka://logging_spark_bdap_dev]
group_id = splunk_dev
index = main
sourcetype = kafka
topic_name = logging_spark_bdap_dev
zookeeper_connect_chroot = kafka-development
zookeeper_connect_host = zookeeper-1.x.x.x
zookeeper_connect_port = 2181
zookeeper_session_timeout_ms = 5000

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...