MQTT Modular Input : All inputs disabled.

[nageshs]

Given that, there are number of posts in this theme, I am probably doing Splunk right. 🙂

Here is the log from splunkd.log

02-06-2015 16:43:23.737 +0530 INFO  SpecFiles - Found external scheme definition for stanza "mqtt://" with 15 parameters: topic_name, broker_host, broker_port, use_ssl, username, passwor
d, client_id, qos, reliable_delivery_dir, clean_session, connection_timeout, keepalive_interval, message_handler_impl, message_handler_params, additional_jvm_propertys
02-06-2015 16:43:23.737 +0530 INFO  SpecFiles - Found external scheme definition for stanza "perfmon://" with 11 parameters: object, counters, instances, interval, mode, samplingInterval
, stats, disabled, index, showZeroValue, useEnglishOnly
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "additional_jvm_propertys":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "broker_host":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "broker_port":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "clean_session":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "client_id":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "connection_timeout":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "keepalive_interval":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "message_handler_impl":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "message_handler_params":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "name":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "password":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "qos":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "reliable_delivery_dir":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "topic_name":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "use_ssl":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "username":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Introspection setup completed for scheme "mqtt".

followed, later, by:

02-06-2015 16:43:32.296 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py" Can't connect to Splunk REST API with the token [Splunk UCsOBihUByd91WO
^mn3L8noZwWJDbpkwcY7Hvf2EWv2W^C0YLBPU95xsleT0Kp4UhWkxRo3Xw_urAqala45dhR8L8gaGad7FoFr8OKNux3mLH_xFCB2WPRN], either the token is invalid or SplunkD has exited : No appropriate protocol (pr
otocol is disabled or cipher suites are inappropriate)
02-06-2015 16:43:42.298 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py" It has been determined via the REST API that all inputs have been disab
led

Going by a similar post around JMX, I checked the JAVA_HOME variable - looks ok.

champ@champ-vm:~$ java -version
java version "1.8.0_31"
Java(TM) SE Runtime Environment (build 1.8.0_31-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.31-b07, mixed mode)
champ@champ-vm:~$ echo $JAVA_HOME
/usr/lib/jvm/java-8-oracle

All said and done, I am pretty sure, this is not a Java problem and in fact, the MQTT connection is probably going on correctly. I tried cloning the MQTT input definition using the Clone link. However, I forgot to change the client identifier. Now, as per the MQTT spec, the client identifier should be unique across a broker. If not, then the formerly connected client will be disconnected. The Eclipse Paho client does provide a call-back upon a disconnect, where, I guess, a re-connect attempt is made. But, since the client ID is same again, now, the recently connected client is forcibly disconnected - trigerring a call back and so on.

I am saying this, because, I did see some exception messages as shown below:

02-06-2015 16:59:46.874 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py" Stanza mqtt://localbroker : Error disconnecting : Client is disconnecte
d (32101)
02-06-2015 16:59:46.874 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at org.eclipse.paho.client.mqttv3.internal.ExceptionHelper.createM
qttException(ExceptionHelper.java:27)
02-06-2015 16:59:46.874 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at org.eclipse.paho.client.mqttv3.internal.ClientComms.disconnect(
ClientComms.java:405)
02-06-2015 16:59:46.874 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at org.eclipse.paho.client.mqttv3.MqttAsyncClient.disconnect(MqttA
syncClient.java:524)
02-06-2015 16:59:46.874 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at org.eclipse.paho.client.mqttv3.MqttClient.disconnect(MqttClient
.java:250)
02-06-2015 16:59:46.874 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at org.eclipse.paho.client.mqttv3.MqttClient.disconnect(MqttClient
.java:243)
02-06-2015 16:59:46.874 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at com.splunk.modinput.mqtt.MQTTModularInput$MessageReceiver.disco
nnect(Unknown Source)
02-06-2015 16:59:46.874 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at com.splunk.modinput.mqtt.MQTTModularInput$MessageReceiver.run(U
nknown Source)
02-06-2015 16:59:46.936 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py" Stanza mqtt://Local : Error running message receiver : Connection lost 
(32109) - java.io.EOFException
02-06-2015 16:59:46.936 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(Comms
Receiver.java:138)
02-06-2015 16:59:46.936 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at java.lang.Thread.run(Thread.java:745)
02-06-2015 16:59:46.936 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py" Caused by: java.io.EOFException
02-06-2015 16:59:46.936 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at java.io.DataInputStream.readByte(DataInputStream.java:267)
02-06-2015 16:59:46.936 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at org.eclipse.paho.client.mqttv3.internal.wire.MqttInputStream.re
adMqttWireMessage(MqttInputStream.java:56)
02-06-2015 16:59:46.936 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(Comms
Receiver.java:100)
02-06-2015 16:59:46.936 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  ... 1 more

The Troubleshooting on the MQTT Modular Input page is not very helpful.

Run this command as the same user that
you are running Splunk as and observe
console output :
"$SPLUNK_HOME/bin/splunk cmd python
../etc/apps/mqtt_ta/bin/mqtt.py
--scheme"

Ok, ran the command and observed - very pretty! Now, what?

<scheme>
    <title>MQTT</title>
    <description>Index messages from a MQTT Broker</description>
    <use_external_validation>true</use_external_validation>
    <streaming_mode>xml</streaming_mode>
    <use_single_instance>true</use_single_instance>
    <endpoint>
        <args>
            <arg name="name">
                <title>Stanza Name</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>true</required_on_create>
            </arg>
            <arg name="topic_name">
                <title>Topic Name</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>true</required_on_create>
            </arg>
            <arg name="broker_host">
                <title>Broker Host</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>true</required_on_create>
            </arg>
            <arg name="broker_port">
                <title>Broker Port</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="use_ssl">
                <title>Use SSL</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="username">
                <title>Username</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="password">
                <title>Password</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="client_id">
                <title>Client ID</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="qos">
                <title>QOS</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="reliable_delivery_dir">
                <title>Reliable Delivery Directory</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="clean_session">
                <title>Clean Session</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="connection_timeout">
                <title>Connection Timeout</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="keepalive_interval">
                <title>Keep Alive Interval</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="additional_jvm_propertys">
                <title>Additional JVM Propertys</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="message_handler_impl">
                <title>Implementation class for a custom message handler</title>
                <description>An implementation of the com.splunk.modinput.mqtt.AbstractMessageHandler class.You would provide this if you required some custom handling/formatting of the messages you consume.Ensure that the necessary jars are in the $SPLUNK_HOME/etc/apps/mqtt_ta/bin/lib directory</description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="message_handler_params">
                <title>Implementation parameter string for the custom message handler</title>
                <description>Parameter string in format 'key1=value1,key2=value2,key3=value3'. This gets passed to the implementation class to process.</description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
        </args>
    </endpoint>
</scheme>

Can you please help me resolve this ?

[Damien_Dallimor]

If you can be a bit patient, it is 1am now for me right now,I can release a TLSv1.2 compatible version 🙂

[nageshs]

... it is 1am now for me right now ...
No, please! Just an assurance, that this is being looked into is more than enough.

BTW, as per the link here, SSLv3 seems to be enabled.

champ@champ-vm:~$ sudo openssl s_client -connect localhost:8000 -ssl3
CONNECTED(00000003)
140032189802144:error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number:s3_pkt.c:337:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 5 bytes and written 7 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : SSLv3
    Cipher    : 0000
    Session-ID: 
    Session-ID-ctx: 
    Master-Key: 
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1423244378
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
---

And yet, splunkd log shows the error.

[Damien_Dallimor]

SSLv3 is disabled by default in Java 8. The MQTT Mod Input uses the Splunk Java SDK under the hood for some things which is hardwired to use SSLv3 to connect to SplunkD.

So use another JRE or enable SSLv3 in your Java 8 runtime.

Side note : at some point(ie: when I get time) I will update all my Java based Modular Inputs to allow for dynamic configuration of the underlying security transport (SSL or TLS) with a customized version of the Splunk Java SDK. That should sort out the errors of this type "No appropriate protocol".

[nageshs]

O.k. let me try OpenJDK (assuming it has enabled SSLv3). Then, step 8 onwards in [this tutorial][1], I could see reports from MQTT publications, I suppose?

tutorial - http://docs.splunk.com/Documentation/Splunk/6.2.1/PivotTutorial/GetthetutorialdataintoSplunk

[nageshs]

So use another JRE or enable SSLv3 in
your Java 8 runtime.

Pretty much every JRE these days disables SSLv3 by default -tested as so with Oracle Java 8, OpenJDK-6 and 7. I set the JVM properties as 'https.protocols=SSLv3' when I created a new stanza for MQTT Input - no luck. Apparently, the override does not seem to take effect.

Too bad, I would have to drop Splunk off my list. :-S