topic Re: MQTT Modular Input : All inputs disabled. in All Apps and Add-ons

MQTT Modular Input : All inputs disabled.

nageshs — Mon, 28 Sep 2020 18:51:59 GMT

Given that, there are number of posts in this theme, I am probably doing Splunk right. 🙂

Here is the log from splunkd.log

02-06-2015 16:43:23.737 +0530 INFO  SpecFiles - Found external scheme definition for stanza "mqtt://" with 15 parameters: topic_name, broker_host, broker_port, use_ssl, username, passwor
d, client_id, qos, reliable_delivery_dir, clean_session, connection_timeout, keepalive_interval, message_handler_impl, message_handler_params, additional_jvm_propertys
02-06-2015 16:43:23.737 +0530 INFO  SpecFiles - Found external scheme definition for stanza "perfmon://" with 11 parameters: object, counters, instances, interval, mode, samplingInterval
, stats, disabled, index, showZeroValue, useEnglishOnly
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "additional_jvm_propertys":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "broker_host":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "broker_port":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "clean_session":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "client_id":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "connection_timeout":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "keepalive_interval":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "message_handler_impl":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "message_handler_params":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "name":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "password":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "qos":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "reliable_delivery_dir":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "topic_name":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "use_ssl":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Endpoint argument settings for "username":
02-06-2015 16:43:24.649 +0530 INFO  ModularInputs - Introspection setup completed for scheme "mqtt".

followed, later, by:

02-06-2015 16:43:32.296 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py" Can't connect to Splunk REST API with the token [Splunk UCsOBihUByd91WO
^mn3L8noZwWJDbpkwcY7Hvf2EWv2W^C0YLBPU95xsleT0Kp4UhWkxRo3Xw_urAqala45dhR8L8gaGad7FoFr8OKNux3mLH_xFCB2WPRN], either the token is invalid or SplunkD has exited : No appropriate protocol (pr
otocol is disabled or cipher suites are inappropriate)
02-06-2015 16:43:42.298 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py" It has been determined via the REST API that all inputs have been disab
led

Going by a similar post around JMX, I checked the JAVA_HOME variable - looks ok.

champ@champ-vm:~$ java -version
java version "1.8.0_31"
Java(TM) SE Runtime Environment (build 1.8.0_31-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.31-b07, mixed mode)
champ@champ-vm:~$ echo $JAVA_HOME
/usr/lib/jvm/java-8-oracle

All said and done, I am pretty sure, this is not a Java problem and in fact, the MQTT connection is probably going on correctly. I tried cloning the MQTT input definition using the Clone link. However, I forgot to change the client identifier. Now, as per the MQTT spec, the client identifier should be unique across a broker. If not, then the formerly connected client will be disconnected. The Eclipse Paho client does provide a call-back upon a disconnect, where, I guess, a re-connect attempt is made. But, since the client ID is same again, now, the recently connected client is forcibly disconnected - trigerring a call back and so on.

I am saying this, because, I did see some exception messages as shown below:

02-06-2015 16:59:46.874 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py" Stanza mqtt://localbroker : Error disconnecting : Client is disconnecte
d (32101)
02-06-2015 16:59:46.874 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at org.eclipse.paho.client.mqttv3.internal.ExceptionHelper.createM
qttException(ExceptionHelper.java:27)
02-06-2015 16:59:46.874 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at org.eclipse.paho.client.mqttv3.internal.ClientComms.disconnect(
ClientComms.java:405)
02-06-2015 16:59:46.874 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at org.eclipse.paho.client.mqttv3.MqttAsyncClient.disconnect(MqttA
syncClient.java:524)
02-06-2015 16:59:46.874 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at org.eclipse.paho.client.mqttv3.MqttClient.disconnect(MqttClient
.java:250)
02-06-2015 16:59:46.874 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at org.eclipse.paho.client.mqttv3.MqttClient.disconnect(MqttClient
.java:243)
02-06-2015 16:59:46.874 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at com.splunk.modinput.mqtt.MQTTModularInput$MessageReceiver.disco
nnect(Unknown Source)
02-06-2015 16:59:46.874 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at com.splunk.modinput.mqtt.MQTTModularInput$MessageReceiver.run(U
nknown Source)
02-06-2015 16:59:46.936 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py" Stanza mqtt://Local : Error running message receiver : Connection lost 
(32109) - java.io.EOFException
02-06-2015 16:59:46.936 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(Comms
Receiver.java:138)
02-06-2015 16:59:46.936 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at java.lang.Thread.run(Thread.java:745)
02-06-2015 16:59:46.936 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py" Caused by: java.io.EOFException
02-06-2015 16:59:46.936 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at java.io.DataInputStream.readByte(DataInputStream.java:267)
02-06-2015 16:59:46.936 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at org.eclipse.paho.client.mqttv3.internal.wire.MqttInputStream.re
adMqttWireMessage(MqttInputStream.java:56)
02-06-2015 16:59:46.936 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(Comms
Receiver.java:100)
02-06-2015 16:59:46.936 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py"  ... 1 more

The Troubleshooting on the MQTT Modular Input page is not very helpful.

Run this command as the same user that
you are running Splunk as and observe
console output :
"$SPLUNK_HOME/bin/splunk cmd python
../etc/apps/mqtt_ta/bin/mqtt.py
--scheme"

Ok, ran the command and observed - very pretty! Now, what?

<scheme>
    <title>MQTT</title>
    <description>Index messages from a MQTT Broker</description>
    <use_external_validation>true</use_external_validation>
    <streaming_mode>xml</streaming_mode>
    <use_single_instance>true</use_single_instance>
    <endpoint>
        <args>
            <arg name="name">
                <title>Stanza Name</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>true</required_on_create>
            </arg>
            <arg name="topic_name">
                <title>Topic Name</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>true</required_on_create>
            </arg>
            <arg name="broker_host">
                <title>Broker Host</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>true</required_on_create>
            </arg>
            <arg name="broker_port">
                <title>Broker Port</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="use_ssl">
                <title>Use SSL</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="username">
                <title>Username</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="password">
                <title>Password</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="client_id">
                <title>Client ID</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="qos">
                <title>QOS</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="reliable_delivery_dir">
                <title>Reliable Delivery Directory</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="clean_session">
                <title>Clean Session</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="connection_timeout">
                <title>Connection Timeout</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="keepalive_interval">
                <title>Keep Alive Interval</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="additional_jvm_propertys">
                <title>Additional JVM Propertys</title>
                <description></description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="message_handler_impl">
                <title>Implementation class for a custom message handler</title>
                <description>An implementation of the com.splunk.modinput.mqtt.AbstractMessageHandler class.You would provide this if you required some custom handling/formatting of the messages you consume.Ensure that the necessary jars are in the $SPLUNK_HOME/etc/apps/mqtt_ta/bin/lib directory</description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="message_handler_params">
                <title>Implementation parameter string for the custom message handler</title>
                <description>Parameter string in format 'key1=value1,key2=value2,key3=value3'. This gets passed to the implementation class to process.</description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
        </args>
    </endpoint>
</scheme>

Can you please help me resolve this ?

Re: MQTT Modular Input : All inputs disabled.

Damien_Dallimor — Fri, 06 Feb 2015 11:53:18 GMT

SSLv3 is disabled by default in Java 8. The MQTT Mod Input uses the Splunk Java SDK under the hood for some things which is hardwired to use SSLv3 to connect to SplunkD.

So use another JRE or enable SSLv3 in your Java 8 runtime.

Side note : at some point(ie: when I get time) I will update all my Java based Modular Inputs to allow for dynamic configuration of the underlying security transport (SSL or TLS) with a customized version of the Splunk Java SDK. That should sort out the errors of this type "No appropriate protocol".

Re: MQTT Modular Input : All inputs disabled.

nageshs — Fri, 06 Feb 2015 13:20:08 GMT

O.k. let me try OpenJDK (assuming it has enabled SSLv3). Then, step 8 onwards in [this tutorial][1], I could see reports from MQTT publications, I suppose?

tutorial - http://docs.splunk.com/Documentation/Splunk/6.2.1/PivotTutorial/GetthetutorialdataintoSplunk

Re: MQTT Modular Input : All inputs disabled.

nageshs — Fri, 06 Feb 2015 17:32:03 GMT

So use another JRE or enable SSLv3 in
your Java 8 runtime.

Pretty much every JRE these days disables SSLv3 by default -tested as so with Oracle Java 8, OpenJDK-6 and 7. I set the JVM properties as 'https.protocols=SSLv3' when I created a new stanza for MQTT Input - no luck. Apparently, the override does not seem to take effect.

Too bad, I would have to drop Splunk off my list. :-S

Re: MQTT Modular Input : All inputs disabled.

Damien_Dallimor — Fri, 06 Feb 2015 17:36:54 GMT

If you can be a bit patient, it is 1am now for me right now,I can release a TLSv1.2 compatible version 🙂

Re: MQTT Modular Input : All inputs disabled.

nageshs — Fri, 06 Feb 2015 17:47:03 GMT

... it is 1am now for me right now ...
No, please! Just an assurance, that this is being looked into is more than enough.

BTW, as per the link here, SSLv3 seems to be enabled.

champ@champ-vm:~$ sudo openssl s_client -connect localhost:8000 -ssl3
CONNECTED(00000003)
140032189802144:error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number:s3_pkt.c:337:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 5 bytes and written 7 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : SSLv3
    Cipher    : 0000
    Session-ID: 
    Session-ID-ctx: 
    Master-Key: 
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1423244378
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
---

And yet, splunkd log shows the error.

Re: MQTT Modular Input : All inputs disabled.

Damien_Dallimor — Sun, 08 Feb 2015 17:21:44 GMT

The latest release has TLSv1.2 support wired in. View the release notes for the latest release for how to enable TLS.

Re: MQTT Modular Input : All inputs disabled.

nageshs — Sun, 08 Feb 2015 18:48:13 GMT

Thanks!

I removed the older package and copied (unzipped) the recent one. I also put the flag value as mentioned in the documentation for the additional JVM values. But, I don't know, what else am I missing...the error persists.

Something you could suggest, please ?

Re: MQTT Modular Input : All inputs disabled.

Damien_Dallimor — Sun, 08 Feb 2015 18:56:49 GMT

It works fine for me with multiple JVM variants.

So I am totally guessing here regarding your predicament....perhaps TLS is not enabled in your Splunk server correctly.

Re: MQTT Modular Input : All inputs disabled.

nageshs — Sun, 08 Feb 2015 19:07:13 GMT

perhaps TLS is not enabled in your
Splunk server correctly.

How does one check that? I do recall seeing some log messages around protocol (SSL, TLS, etc.) support at server.

Re: MQTT Modular Input : All inputs disabled.

Damien_Dallimor — Sun, 08 Feb 2015 19:29:08 GMT

http://docs.splunk.com/Documentation/Splunk/6.2.1/Security/SetyourSSLversion

Re: MQTT Modular Input : All inputs disabled.

nageshs — Mon, 09 Feb 2015 08:34:01 GMT

This is what I have inweb.conf. Still the same error.

# Allow only SSLv3 and above connections by default
sslVersions = ssl3, tls

Re: MQTT Modular Input : All inputs disabled.

nageshs — Mon, 09 Feb 2015 10:03:42 GMT

I don't know what to say....

I have fresh Ubuntu-14.04 VM image, JAVA_HOME set to /usr/lib/jvm/java-8-oracle, installed Splunk, installed MQTT input, set the additional JVM parameters to splunk.securetransport.protocol=tls. But, I still see this error message.

02-09-2015 15:19:30.709 +0530 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/mqtt_ta/bin/mqtt.py" Can't connect to Splunk REST API with the token [Splunk CqQga73SxlT6IrU
FaZBxGkKApir_1Yg4bYrY8QN0hchCOhddr1rANljEICVE4xLp0PCodzQAC3XPRC^J52^pRGNFezlSgBz8t8KKKtWgv7R652vjXQ^IzGxxgoC], either the token is invalid or SplunkD has exited : No appropriate protocol
 (protocol is disabled or cipher suites are inappropriate)

Re: MQTT Modular Input : All inputs disabled.

Damien_Dallimor — Mon, 09 Feb 2015 10:41:33 GMT

check your SSL/TLS settings in server.conf

also , what does your inputs.conf stanza for your MQTT setup currently look like ?

Re: MQTT Modular Input : All inputs disabled.

nageshs — Mon, 09 Feb 2015 13:41:17 GMT

server.conf

[sslConfig]
enableSplunkdSSL = true
useClientSSLCompression = true
useSplunkdClientSSLCompression = true 
# enableSplunkSearchSSL has been moved to web.conf,  and changed to enableSplunkWebSSL

#Allow only sslv3 and above connections to the HTTP server
sslVersions = *,-ssl2
sendStrictTransportSecurityHeader = false
allowSslCompression = true
allowSslRenegotiation = true

# For the HTTP server, Diable ciphers lower than 128-bit and disallow ciphers that
# don't provide authentication and/or encryption.
# Use 'openssl ciphers -v' to generate a list of supported ciphers
# Allow only TLSv1 cipher with 'high' encryption suits, i.e. whose key lengths are
# larger than or equal to 128 bits
cipherSuite = TLSv1+HIGH:@STRENGTH

sslKeysfile = server.pem
sslKeysfilePassword = password
caCertFile = cacert.pem
caPath = $SPLUNK_HOME/etc/auth
certCreateScript = $SPLUNK_HOME/bin/splunk, createssl, server-cert

web.conf

# SSL certificate files. Paths are specified relative to SPLUNK_HOME
privKeyPath = etc/auth/splunkweb/privkey.pem
caCertPath = etc/auth/splunkweb/cert.pem

# Allow only SSLv3 and above connections by default
sslVersions = ssl3, tls

Re: MQTT Modular Input : All inputs disabled.

nageshs — Mon, 09 Feb 2015 13:46:20 GMT

The inputs.conf does not have anything w.r.t MQTT; but, I did add a data input from the web page. I can see some entries in log for the same.

02-09-2015 15:19:22.338 +0530 INFO  SpecFiles - Found external scheme definition for stanza "mqtt://" with 15 parameters: topic_name, broker_host, broker_port, use_ssl, username, passwor
d, client_id, qos, reliable_delivery_dir, clean_session, connection_timeout, keepalive_interval, message_handler_impl, message_handler_params, additional_jvm_propertys

Few lines (from below) removed for brevity ...

02-09-2015 15:19:22.847 +0530 INFO  ModularInputs - Endpoint argument settings for "additional_jvm_propertys":
02-09-2015 15:19:22.847 +0530 INFO  ModularInputs - Endpoint argument settings for "broker_host":
02-09-2015 15:19:22.847 +0530 INFO  ModularInputs - Endpoint argument settings for "broker_port":
02-09-2015 15:19:22.847 +0530 INFO  ModularInputs - Endpoint argument settings for "clean_session":

Re: MQTT Modular Input : All inputs disabled.

Damien_Dallimor — Mon, 09 Feb 2015 13:52:31 GMT

inputs.conf is where your MQTT stanza gets persisted to when you enter the values via the web interface.

I need to see this If I am to verify your setup.

Re: MQTT Modular Input : All inputs disabled.

nageshs — Mon, 09 Feb 2015 14:26:37 GMT

Since I don't know why the stanza wouldn't persist, can you show me how do the name-value pairs look like? I will edit directly into the file.

BTW, I am logging with my user ID and starting the browser with admin user ID. I can see a user id with name as splunk created; but, I am unable to start the server with this id (permissions problem).

Re: MQTT Modular Input : All inputs disabled.

Damien_Dallimor — Mon, 09 Feb 2015 15:31:37 GMT

You have not understood me.

I want to see your inputs.conf MQTT stanza to verify that you setup MQTT correctly.

If you don't know where to find the inputs.conf file , then try some command line searching :

Change to your Splunk home directory , then :

find . -name "inputs.conf" | xargs grep mqtt

Re: MQTT Modular Input : All inputs disabled.

nageshs — Mon, 09 Feb 2015 16:36:14 GMT

Thanks for the tip. Here is the result.

champ@champ-vm:/opt/splunk$ sudo find . -name "inputs.conf" | sudo xargs grep mqtt
./etc/apps/launcher/local/inputs.conf:[mqtt://Local]
champ@champ-vm:/opt/splunk$ sudo more etc/apps/launcher/local/inputs.conf
[mqtt://Local]
additional_jvm_propertys = 'splunk.securetransport.protocol=tls'
broker_host = localhost
broker_port = 1883
clean_session = 0
qos = 0
sourcetype = csv
topic_name = readings/temperature
use_ssl = 0

BTW, I had tried the value of additional_jvm_propertys as splunk.securetransport.protocol=tls with and without quotes.