MQTT Modular Input : All inputs disabled.

[nageshs]

Given that, there are number of posts in this theme, I am probably doing Splunk right. 🙂

Here is the log from splunkd.log

02-06-2015 16:43:23.737 +0530 INFO  SpecFiles - Found external scheme definition for stanza "mqtt://" with 15 parameters: topic_name, broker_host, broker_port, use_ssl, username, passwor
d, client_id, qos, reliable_delivery_dir, clean_session, connection_timeout, keepalive_interval, message_handler_impl, message_handler_params, additional_jvm_propertys
02-06-2015 16:43:23.737 +0530 INFO  SpecFiles - Found external scheme definition for stanza "perfmon://" with 11 parameters: object, counters, instances, interval, mode, samplingInterval
, stats, disabled, index, showZeroValue, useEnglishOnly
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "additional_jvm_propertys":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "broker_host":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "broker_port":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "clean_session":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "client_id":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "connection_timeout":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "keepalive_interval":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "message_handler_impl":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "message_handler_params":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "name":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "password":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "qos":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "reliable_delivery_dir":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "topic_name":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "use_ssl":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "username":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Introspection setup completed for scheme "mqtt".

followed, later, by:

02-06-2015 16:43:32.296 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py" Can't connect to Splunk REST API with the token [Splunk UCsOBihUByd91WO
^mn3L8noZwWJDbpkwcY7Hvf2EWv2W^C0YLBPU95xsleT0Kp4UhWkxRo3Xw_urAqala45dhR8L8gaGad7FoFr8OKNux3mLH_xFCB2WPRN], either the token is invalid or SplunkD has exited : No appropriate protocol (pr
otocol is disabled or cipher suites are inappropriate)
02-06-2015 16:43:42.298 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py" It has been determined via the REST API that all inputs have been disab
led

Going by a similar post around JMX, I checked the JAVA_HOME variable - looks ok.

champ@champ-vm:~$ java -version
java version "1.8.0_31"
Java(TM) SE Runtime Environment (build 1.8.0_31-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.31-b07, mixed mode)
champ@champ-vm:~$ echo $JAVA_HOME
/usr/lib/jvm/java-8-oracle

All said and done, I am pretty sure, this is not a Java problem and in fact, the MQTT connection is probably going on correctly. I tried cloning the MQTT input definition using the Clone link. However, I forgot to change the client identifier. Now, as per the MQTT spec, the client identifier should be unique across a broker. If not, then the formerly connected client will be disconnected. The Eclipse Paho client does provide a call-back upon a disconnect, where, I guess, a re-connect attempt is made. But, since the client ID is same again, now, the recently connected client is forcibly disconnected - trigerring a call back and so on.

I am saying this, because, I did see some exception messages as shown below:

02-06-2015 16:59:46.874 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py" Stanza mqtt://localbroker : Error disconnecting : Client is disconnecte
d (32101)
02-06-2015 16:59:46.874 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at org.eclipse.paho.client.mqttv3.internal.ExceptionHelper.createM
qttException(ExceptionHelper.java:27)
02-06-2015 16:59:46.874 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at org.eclipse.paho.client.mqttv3.internal.ClientComms.disconnect(
ClientComms.java:405)
02-06-2015 16:59:46.874 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at org.eclipse.paho.client.mqttv3.MqttAsyncClient.disconnect(MqttA
syncClient.java:524)
02-06-2015 16:59:46.874 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at org.eclipse.paho.client.mqttv3.MqttClient.disconnect(MqttClient
.java:250)
02-06-2015 16:59:46.874 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at org.eclipse.paho.client.mqttv3.MqttClient.disconnect(MqttClient
.java:243)
02-06-2015 16:59:46.874 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at com.splunk.modinput.mqtt.MQTTModularInput$MessageReceiver.disco
nnect(Unknown Source)
02-06-2015 16:59:46.874 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at com.splunk.modinput.mqtt.MQTTModularInput$MessageReceiver.run(U
nknown Source)
02-06-2015 16:59:46.936 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py" Stanza mqtt://Local : Error running message receiver : Connection lost 
(32109) - java.io.EOFException
02-06-2015 16:59:46.936 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(Comms
Receiver.java:138)
02-06-2015 16:59:46.936 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at java.lang.Thread.run(Thread.java:745)
02-06-2015 16:59:46.936 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py" Caused by: java.io.EOFException
02-06-2015 16:59:46.936 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at java.io.DataInputStream.readByte(DataInputStream.java:267)
02-06-2015 16:59:46.936 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at org.eclipse.paho.client.mqttv3.internal.wire.MqttInputStream.re
adMqttWireMessage(MqttInputStream.java:56)
02-06-2015 16:59:46.936 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(Comms
Receiver.java:100)
02-06-2015 16:59:46.936 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  ... 1 more

The Troubleshooting on the MQTT Modular Input page is not very helpful.

Run this command as the same user that
you are running Splunk as and observe
console output :
"$SPLUNK_HOME/bin/splunk cmd python
../etc/apps/mqtt_ta/bin/mqtt.py
--scheme"

Ok, ran the command and observed - very pretty! Now, what?

<scheme>
    <title>MQTT</title>
    <description>Index messages from a MQTT Broker</description>
    <use_external_validation>true</use_external_validation>
    <streaming_mode>xml</streaming_mode>
    <use_single_instance>true</use_single_instance>
    <endpoint>
        <args>
            <arg name="name">
                <title>Stanza Name</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>true</required_on_create>
            </arg>
            <arg name="topic_name">
                <title>Topic Name</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>true</required_on_create>
            </arg>
            <arg name="broker_host">
                <title>Broker Host</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>true</required_on_create>
            </arg>
            <arg name="broker_port">
                <title>Broker Port</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="use_ssl">
                <title>Use SSL</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="username">
                <title>Username</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="password">
                <title>Password</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="client_id">
                <title>Client ID</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="qos">
                <title>QOS</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="reliable_delivery_dir">
                <title>Reliable Delivery Directory</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="clean_session">
                <title>Clean Session</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="connection_timeout">
                <title>Connection Timeout</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="keepalive_interval">
                <title>Keep Alive Interval</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="additional_jvm_propertys">
                <title>Additional JVM Propertys</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="message_handler_impl">
                <title>Implementation class for a custom message handler</title>
                <description>An implementation of the com.splunk.modinput.mqtt.AbstractMessageHandler class.You would provide this if you required some custom handling/formatting of the messages you consume.Ensure that the necessary jars are in the $SPLUNK_HOME/etc/apps/mqtt_ta/bin/lib directory</description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="message_handler_params">
                <title>Implementation parameter string for the custom message handler</title>
                <description>Parameter string in format 'key1=value1,key2=value2,key3=value3'. This gets passed to the implementation class to process.</description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
        </args>
    </endpoint>
</scheme>

Can you please help me resolve this ?

[Damien_Dallimor · Accepted solution]

The latest release has TLSv1.2 support wired in. View the release notes for the latest release for how to enable TLS.

[nageshs]

Thanks!

I removed the older package and copied (unzipped) the recent one. I also put the flag value as mentioned in the documentation for the additional JVM values. But, I don't know, what else am I missing...the error persists.

Something you could suggest, please ?

[Damien_Dallimor]

It works fine for me with multiple JVM variants.

So I am totally guessing here regarding your predicament....perhaps TLS is not enabled in your Splunk server correctly.

[nageshs]

perhaps TLS is not enabled in your
Splunk server correctly.

How does one check that? I do recall seeing some log messages around protocol (SSL, TLS, etc.) support at server.

[Damien_Dallimor]

http://docs.splunk.com/Documentation/Splunk/6.2.1/Security/SetyourSSLversion

[nageshs]

This is what I have inweb.conf. Still the same error.

# Allow only SSLv3 and above connections by default
sslVersions = ssl3, tls

[Damien_Dallimor]

check your SSL/TLS settings in server.conf

also , what does your inputs.conf stanza for your MQTT setup currently look like ?

[nageshs]

server.conf

[sslConfig]
enableSplunkdSSL = true
useClientSSLCompression = true
useSplunkdClientSSLCompression = true 
# enableSplunkSearchSSL has been moved to web.conf,  and changed to enableSplunkWebSSL

#Allow only sslv3 and above connections to the HTTP server
sslVersions = *,-ssl2
sendStrictTransportSecurityHeader = false
allowSslCompression = true
allowSslRenegotiation = true

# For the HTTP server, Diable ciphers lower than 128-bit and disallow ciphers that
# don't provide authentication and/or encryption.
# Use 'openssl ciphers -v' to generate a list of supported ciphers
# Allow only TLSv1 cipher with 'high' encryption suits, i.e. whose key lengths are
# larger than or equal to 128 bits
cipherSuite = TLSv1+HIGH:@STRENGTH

sslKeysfile = server.pem
sslKeysfilePassword = password
caCertFile = cacert.pem
caPath = $SPLUNK_HOME/etc/auth
certCreateScript = $SPLUNK_HOME/bin/splunk, createssl, server-cert

web.conf

# SSL certificate files. Paths are specified relative to SPLUNK_HOME
privKeyPath = etc/auth/splunkweb/privkey.pem
caCertPath = etc/auth/splunkweb/cert.pem

# Allow only SSLv3 and above connections by default
sslVersions = ssl3, tls

[nageshs]

The inputs.conf does not have anything w.r.t MQTT; but, I did add a data input from the web page. I can see some entries in log for the same.

02-09-2015 15:19:22.338 +0530 INFO  SpecFiles - Found external scheme definition for stanza "mqtt://" with 15 parameters: topic_name, broker_host, broker_port, use_ssl, username, passwor
d, client_id, qos, reliable_delivery_dir, clean_session, connection_timeout, keepalive_interval, message_handler_impl, message_handler_params, additional_jvm_propertys

Few lines (from below) removed for brevity ...

02-09-2015 15:19:22.847 +0530 INFO  ModularInputs - Endpoint argument settings for "additional_jvm_propertys":
02-09-2015 15:19:22.847 +0530 INFO  ModularInputs - Endpoint argument settings for "broker_host":
02-09-2015 15:19:22.847 +0530 INFO  ModularInputs - Endpoint argument settings for "broker_port":
02-09-2015 15:19:22.847 +0530 INFO  ModularInputs - Endpoint argument settings for "clean_session":

[Damien_Dallimor]

inputs.conf is where your MQTT stanza gets persisted to when you enter the values via the web interface.

I need to see this If I am to verify your setup.

[nageshs]

Since I don't know why the stanza wouldn't persist, can you show me how do the name-value pairs look like? I will edit directly into the file.

BTW, I am logging with my user ID and starting the browser with admin user ID. I can see a user id with name as splunk created; but, I am unable to start the server with this id (permissions problem).

[Damien_Dallimor]

You have not understood me.

I want to see your inputs.conf MQTT stanza to verify that you setup MQTT correctly.

If you don't know where to find the inputs.conf file , then try some command line searching :

Change to your Splunk home directory , then :

find . -name "inputs.conf" | xargs grep mqtt

[nageshs]

Thanks for the tip. Here is the result.

champ@champ-vm:/opt/splunk$ sudo find . -name "inputs.conf" | sudo xargs grep mqtt
./etc/apps/launcher/local/inputs.conf:[mqtt://Local]
champ@champ-vm:/opt/splunk$ sudo more etc/apps/launcher/local/inputs.conf
[mqtt://Local]
additional_jvm_propertys = 'splunk.securetransport.protocol=tls'
broker_host = localhost
broker_port = 1883
clean_session = 0
qos = 0
sourcetype = csv
topic_name = readings/temperature
use_ssl = 0

BTW, I had tried the value of additional_jvm_propertys as splunk.securetransport.protocol=tls with and without quotes.

[Damien_Dallimor]

No quotes required.

What files are in your etc/apps/mqtt_ta/bin/lib directory ?

Checking you upgraded correctly.

[nageshs]

Quotes removed and restarted - no change.

These are the files in the said location.

champ@champ-vm:~$ cd /opt/splunk/etc/apps/mqtt_ta/bin/lib
champ@champ-vm:/opt/splunk/etc/apps/mqtt_ta/bin/lib$ ls -l
total 908
-rw-r--r-- 1 splunk splunk 481534 Feb  8 21:19 log4j-1.2.16.jar
-rw-r--r-- 1 splunk splunk 146728 Feb  8 21:19 mqtt-client-0.4.0.jar
-rw-r--r-- 1 splunk splunk  36497 Feb  8 21:19 mqttmodinput.jar
-rw-r--r-- 1 splunk splunk 259633 Feb  8 21:19 splunk_tlsv12.jar

[Damien_Dallimor]

Think I found a bug.

So I rolled back the newest release.

Back to having to enable SSLv3 in your Java 8 environment.

[nageshs]

ok, you might also want to use a more recent MQTT client from Paho - [Paho MQTTv3 client][1].

https://repo.eclipse.org/content/repositories/paho-releases/org/eclipse/paho/org.eclipse.paho.client...

[Damien_Dallimor]

Update : try the version here : http://damiendallimore.github.io/ (before I release it and screw up Splunkbase again 🙂 )

I weeded out the TLS bug / SDK incompatibilitys issues

[nageshs]

It is working now ! Many thanks, Damien !!

[nageshs]

I don't know what to say....

I have fresh Ubuntu-14.04 VM image, JAVA_HOME set to /usr/lib/jvm/java-8-oracle, installed Splunk, installed MQTT input, set the additional JVM parameters to splunk.securetransport.protocol=tls. But, I still see this error message.

02-09-2015 15:19:30.709 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py" Can't connect to Splunk REST API with the token [Splunk CqQga73SxlT6IrU
FaZBxGkKApir_1Yg4bYrY8QN0hchCOhddr1rANljEICVE4xLp0PCodzQAC3XPRC^J52^pRGNFezlSgBz8t8KKKtWgv7R652vjXQ^IzGxxgoC], either the token is invalid or SplunkD has exited : No appropriate protocol
 (protocol is disabled or cipher suites are inappropriate)