All Apps and Add-ons

MQTT Modular Input : All inputs disabled.

nageshs
Engager

Given that, there are number of posts in this theme, I am probably doing Splunk right. 🙂

Here is the log from splunkd.log

02-06-2015 16:43:23.737 +0530 INFO  SpecFiles - Found external scheme definition for stanza "mqtt://" with 15 parameters: topic_name, broker_host, broker_port, use_ssl, username, passwor
d, client_id, qos, reliable_delivery_dir, clean_session, connection_timeout, keepalive_interval, message_handler_impl, message_handler_params, additional_jvm_propertys
02-06-2015 16:43:23.737 +0530 INFO  SpecFiles - Found external scheme definition for stanza "perfmon://" with 11 parameters: object, counters, instances, interval, mode, samplingInterval
, stats, disabled, index, showZeroValue, useEnglishOnly
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "additional_jvm_propertys":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "broker_host":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "broker_port":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "clean_session":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "client_id":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "connection_timeout":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "keepalive_interval":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "message_handler_impl":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "message_handler_params":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "name":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "password":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "qos":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "reliable_delivery_dir":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "topic_name":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "use_ssl":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "username":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Introspection setup completed for scheme "mqtt".

followed, later, by:

02-06-2015 16:43:32.296 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py" Can't connect to Splunk REST API with the token [Splunk UCsOBihUByd91WO
^mn3L8noZwWJDbpkwcY7Hvf2EWv2W^C0YLBPU95xsleT0Kp4UhWkxRo3Xw_urAqala45dhR8L8gaGad7FoFr8OKNux3mLH_xFCB2WPRN], either the token is invalid or SplunkD has exited : No appropriate protocol (pr
otocol is disabled or cipher suites are inappropriate)
02-06-2015 16:43:42.298 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py" It has been determined via the REST API that all inputs have been disab
led

Going by a similar post around JMX, I checked the JAVA_HOME variable - looks ok.

champ@champ-vm:~$ java -version
java version "1.8.0_31"
Java(TM) SE Runtime Environment (build 1.8.0_31-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.31-b07, mixed mode)
champ@champ-vm:~$ echo $JAVA_HOME
/usr/lib/jvm/java-8-oracle

All said and done, I am pretty sure, this is not a Java problem and in fact, the MQTT connection is probably going on correctly. I tried cloning the MQTT input definition using the Clone link. However, I forgot to change the client identifier. Now, as per the MQTT spec, the client identifier should be unique across a broker. If not, then the formerly connected client will be disconnected. The Eclipse Paho client does provide a call-back upon a disconnect, where, I guess, a re-connect attempt is made. But, since the client ID is same again, now, the recently connected client is forcibly disconnected - trigerring a call back and so on.

I am saying this, because, I did see some exception messages as shown below:

02-06-2015 16:59:46.874 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py" Stanza mqtt://localbroker : Error disconnecting : Client is disconnecte
d (32101)
02-06-2015 16:59:46.874 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at org.eclipse.paho.client.mqttv3.internal.ExceptionHelper.createM
qttException(ExceptionHelper.java:27)
02-06-2015 16:59:46.874 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at org.eclipse.paho.client.mqttv3.internal.ClientComms.disconnect(
ClientComms.java:405)
02-06-2015 16:59:46.874 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at org.eclipse.paho.client.mqttv3.MqttAsyncClient.disconnect(MqttA
syncClient.java:524)
02-06-2015 16:59:46.874 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at org.eclipse.paho.client.mqttv3.MqttClient.disconnect(MqttClient
.java:250)
02-06-2015 16:59:46.874 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at org.eclipse.paho.client.mqttv3.MqttClient.disconnect(MqttClient
.java:243)
02-06-2015 16:59:46.874 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at com.splunk.modinput.mqtt.MQTTModularInput$MessageReceiver.disco
nnect(Unknown Source)
02-06-2015 16:59:46.874 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at com.splunk.modinput.mqtt.MQTTModularInput$MessageReceiver.run(U
nknown Source)
02-06-2015 16:59:46.936 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py" Stanza mqtt://Local : Error running message receiver : Connection lost 
(32109) - java.io.EOFException
02-06-2015 16:59:46.936 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(Comms
Receiver.java:138)
02-06-2015 16:59:46.936 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at java.lang.Thread.run(Thread.java:745)
02-06-2015 16:59:46.936 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py" Caused by: java.io.EOFException
02-06-2015 16:59:46.936 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at java.io.DataInputStream.readByte(DataInputStream.java:267)
02-06-2015 16:59:46.936 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at org.eclipse.paho.client.mqttv3.internal.wire.MqttInputStream.re
adMqttWireMessage(MqttInputStream.java:56)
02-06-2015 16:59:46.936 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(Comms
Receiver.java:100)
02-06-2015 16:59:46.936 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  ... 1 more

The Troubleshooting on the MQTT Modular Input page is not very helpful.

Run this command as the same user that
you are running Splunk as and observe
console output :
"$SPLUNK_HOME/bin/splunk cmd python
../etc/apps/mqtt_ta/bin/mqtt.py
--scheme"

Ok, ran the command and observed - very pretty! Now, what?

<scheme>
    <title>MQTT</title>
    <description>Index messages from a MQTT Broker</description>
    <use_external_validation>true</use_external_validation>
    <streaming_mode>xml</streaming_mode>
    <use_single_instance>true</use_single_instance>
    <endpoint>
        <args>
            <arg name="name">
                <title>Stanza Name</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>true</required_on_create>
            </arg>
            <arg name="topic_name">
                <title>Topic Name</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>true</required_on_create>
            </arg>
            <arg name="broker_host">
                <title>Broker Host</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>true</required_on_create>
            </arg>
            <arg name="broker_port">
                <title>Broker Port</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="use_ssl">
                <title>Use SSL</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="username">
                <title>Username</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="password">
                <title>Password</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="client_id">
                <title>Client ID</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="qos">
                <title>QOS</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="reliable_delivery_dir">
                <title>Reliable Delivery Directory</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="clean_session">
                <title>Clean Session</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="connection_timeout">
                <title>Connection Timeout</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="keepalive_interval">
                <title>Keep Alive Interval</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="additional_jvm_propertys">
                <title>Additional JVM Propertys</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="message_handler_impl">
                <title>Implementation class for a custom message handler</title>
                <description>An implementation of the com.splunk.modinput.mqtt.AbstractMessageHandler class.You would provide this if you required some custom handling/formatting of the messages you consume.Ensure that the necessary jars are in the $SPLUNK_HOME/etc/apps/mqtt_ta/bin/lib directory</description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="message_handler_params">
                <title>Implementation parameter string for the custom message handler</title>
                <description>Parameter string in format 'key1=value1,key2=value2,key3=value3'. This gets passed to the implementation class to process.</description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
        </args>
    </endpoint>
</scheme>

Can you please help me resolve this ?

Tags (1)
0 Karma
1 Solution

Damien_Dallimor
Ultra Champion

The latest release has TLSv1.2 support wired in. View the release notes for the latest release for how to enable TLS.

View solution in original post

Damien_Dallimor
Ultra Champion

The latest release has TLSv1.2 support wired in. View the release notes for the latest release for how to enable TLS.

View solution in original post

nageshs
Engager

Thanks!

I removed the older package and copied (unzipped) the recent one. I also put the flag value as mentioned in the documentation for the additional JVM values. But, I don't know, what else am I missing...the error persists.

Something you could suggest, please ?

0 Karma

Damien_Dallimor
Ultra Champion

It works fine for me with multiple JVM variants.

So I am totally guessing here regarding your predicament....perhaps TLS is not enabled in your Splunk server correctly.

0 Karma

nageshs
Engager

perhaps TLS is not enabled in your
Splunk server correctly.

How does one check that? I do recall seeing some log messages around protocol (SSL, TLS, etc.) support at server.

0 Karma

Damien_Dallimor
Ultra Champion
0 Karma

nageshs
Engager

This is what I have inweb.conf. Still the same error.

# Allow only SSLv3 and above connections by default
sslVersions = ssl3, tls
0 Karma

Damien_Dallimor
Ultra Champion

check your SSL/TLS settings in server.conf

also , what does your inputs.conf stanza for your MQTT setup currently look like ?

0 Karma

nageshs
Engager

server.conf

[sslConfig]
enableSplunkdSSL = true
useClientSSLCompression = true
useSplunkdClientSSLCompression = true 
# enableSplunkSearchSSL has been moved to web.conf,  and changed to enableSplunkWebSSL

#Allow only sslv3 and above connections to the HTTP server
sslVersions = *,-ssl2
sendStrictTransportSecurityHeader = false
allowSslCompression = true
allowSslRenegotiation = true

# For the HTTP server, Diable ciphers lower than 128-bit and disallow ciphers that
# don't provide authentication and/or encryption.
# Use 'openssl ciphers -v' to generate a list of supported ciphers
# Allow only TLSv1 cipher with 'high' encryption suits, i.e. whose key lengths are
# larger than or equal to 128 bits
cipherSuite = TLSv1+HIGH:@STRENGTH

sslKeysfile = server.pem
sslKeysfilePassword = password
caCertFile = cacert.pem
caPath = $SPLUNK_HOME/etc/auth
certCreateScript = $SPLUNK_HOME/bin/splunk, createssl, server-cert

web.conf

# SSL certificate files. Paths are specified relative to SPLUNK_HOME
privKeyPath = etc/auth/splunkweb/privkey.pem
caCertPath = etc/auth/splunkweb/cert.pem

# Allow only SSLv3 and above connections by default
sslVersions = ssl3, tls
0 Karma

nageshs
Engager

The inputs.conf does not have anything w.r.t MQTT; but, I did add a data input from the web page. I can see some entries in log for the same.

02-09-2015 15:19:22.338 +0530 INFO  SpecFiles - Found external scheme definition for stanza "mqtt://" with 15 parameters: topic_name, broker_host, broker_port, use_ssl, username, passwor
d, client_id, qos, reliable_delivery_dir, clean_session, connection_timeout, keepalive_interval, message_handler_impl, message_handler_params, additional_jvm_propertys

Few lines (from below) removed for brevity ...

02-09-2015 15:19:22.847 +0530 INFO  ModularInputs - Endpoint argument settings for "additional_jvm_propertys":
02-09-2015 15:19:22.847 +0530 INFO  ModularInputs - Endpoint argument settings for "broker_host":
02-09-2015 15:19:22.847 +0530 INFO  ModularInputs - Endpoint argument settings for "broker_port":
02-09-2015 15:19:22.847 +0530 INFO  ModularInputs - Endpoint argument settings for "clean_session":
0 Karma

Damien_Dallimor
Ultra Champion

inputs.conf is where your MQTT stanza gets persisted to when you enter the values via the web interface.

I need to see this If I am to verify your setup.

0 Karma

nageshs
Engager

Since I don't know why the stanza wouldn't persist, can you show me how do the name-value pairs look like? I will edit directly into the file.

BTW, I am logging with my user ID and starting the browser with admin user ID. I can see a user id with name as splunk created; but, I am unable to start the server with this id (permissions problem).

0 Karma

Damien_Dallimor
Ultra Champion

You have not understood me.

I want to see your inputs.conf MQTT stanza to verify that you setup MQTT correctly.

If you don't know where to find the inputs.conf file , then try some command line searching :

Change to your Splunk home directory , then :

find . -name "inputs.conf" | xargs grep mqtt

0 Karma

nageshs
Engager

Thanks for the tip. Here is the result.

champ@champ-vm:/opt/splunk$ sudo find . -name "inputs.conf" | sudo xargs grep mqtt
./etc/apps/launcher/local/inputs.conf:[mqtt://Local]
champ@champ-vm:/opt/splunk$ sudo more etc/apps/launcher/local/inputs.conf
[mqtt://Local]
additional_jvm_propertys = 'splunk.securetransport.protocol=tls'
broker_host = localhost
broker_port = 1883
clean_session = 0
qos = 0
sourcetype = csv
topic_name = readings/temperature
use_ssl = 0

BTW, I had tried the value of additional_jvm_propertys as splunk.securetransport.protocol=tls with and without quotes.

0 Karma

Damien_Dallimor
Ultra Champion

No quotes required.

What files are in your etc/apps/mqtt_ta/bin/lib directory ?

Checking you upgraded correctly.

0 Karma

nageshs
Engager

Quotes removed and restarted - no change.

These are the files in the said location.

champ@champ-vm:~$ cd /opt/splunk/etc/apps/mqtt_ta/bin/lib
champ@champ-vm:/opt/splunk/etc/apps/mqtt_ta/bin/lib$ ls -l
total 908
-rw-r--r-- 1 splunk splunk 481534 Feb  8 21:19 log4j-1.2.16.jar
-rw-r--r-- 1 splunk splunk 146728 Feb  8 21:19 mqtt-client-0.4.0.jar
-rw-r--r-- 1 splunk splunk  36497 Feb  8 21:19 mqttmodinput.jar
-rw-r--r-- 1 splunk splunk 259633 Feb  8 21:19 splunk_tlsv12.jar
0 Karma

Damien_Dallimor
Ultra Champion

Think I found a bug.

So I rolled back the newest release.

Back to having to enable SSLv3 in your Java 8 environment.

0 Karma

nageshs
Engager

ok, you might also want to use a more recent MQTT client from Paho - [Paho MQTTv3 client][1].

https://repo.eclipse.org/content/repositories/paho-releases/org/eclipse/paho/org.eclipse.paho.client...

0 Karma

Damien_Dallimor
Ultra Champion

Update : try the version here : http://damiendallimore.github.io/ (before I release it and screw up Splunkbase again 🙂 )

I weeded out the TLS bug / SDK incompatibilitys issues

0 Karma

nageshs
Engager

It is working now ! Many thanks, Damien !!

0 Karma

nageshs
Engager

I don't know what to say....

I have fresh Ubuntu-14.04 VM image, JAVA_HOME set to /usr/lib/jvm/java-8-oracle, installed Splunk, installed MQTT input, set the additional JVM parameters to splunk.securetransport.protocol=tls. But, I still see this error message.

02-09-2015 15:19:30.709 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py" Can't connect to Splunk REST API with the token [Splunk CqQga73SxlT6IrU
FaZBxGkKApir_1Yg4bYrY8QN0hchCOhddr1rANljEICVE4xLp0PCodzQAC3XPRC^J52^pRGNFezlSgBz8t8KKKtWgv7R652vjXQ^IzGxxgoC], either the token is invalid or SplunkD has exited : No appropriate protocol
 (protocol is disabled or cipher suites are inappropriate)
0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!