Splunk Add-on for IBM WAS: "Unable to initialize modular input "jmx" defined inside the app "Splunk_TA_jmx": Introspecting scheme=jmx: script running failed"

[glentes]

Dear splunk community,

We get the following error message trying to get SPLUNK_TA_jmx 3.1.0 working on a 64Bit SLES 11 SP3 system with splunk 6.2.1.2 build 259063:

Unable to initialize modular input "jmx" defined inside the app "Splunk_TA_jmx": Introspecting scheme=jmx: script running failed (exited with code 1).

Some background:

We set up SPLUNK_TA_jmx for testing on a Windows server 2008 R2 running Splunk 6.2.4 build 271043 and it works without any problems.
Now we want to install it on a heavy forwarder running on 64Bit SLES 11 SP3 in our production Splunk infrastructure with search head and indexer clustering.
We set up Splunk 6.2.1 as a heavy forwarder with

[indexAndForward]
index = false

[tcpout:default-autolb-group]
server = <list of index servers in the index cluster>

We created an index for the events on the indexer cluster servers and the heavy forwarder.

From the splunkd.log:

12-29-2015 14:52:50.917 +0000 ERROR ModularInputs - Introspecting scheme=jmx: script running failed (exited with code 1).
12-29-2015 14:52:50.918 +0000 ERROR ModularInputs - Unable to initialize modular input "jmx"  defined inside the app "Splunk_TA_jmx": Introspecting scheme=jmx: script running failed (exited with code 1).

We found the following information on answers.splunk.com with similar problems, but they don't solve our issue:

https://answers.splunk.com/answers/236224/how-do-i-troubleshoot-a-problem-with-jmx-add-on.html
https://answers.splunk.com/answers/152040/unable-to-get-jmx-data-with-app-monitoring-of-java-virtual...
https://answers.splunk.com/answers/217742/monitoring-of-java-virtual-machines-with-jmx-how-t-2.html?...
https://answers.splunk.com/answers/118243/unable-to-initialize-modular-input-jmx-defined-inside-the-...

See for example the following output for reference:

splunk@xxxxxx:~/etc/apps/Splunk_TA_jmx> $SPLUNK_HOME/bin/splunk cmd /opt/splunk/bin/python $SPLUNK_HOME/etc/apps/Splunk_TA_jmx/bin/jmx.py --scheme
<scheme>
    <title>JMX (Java Management Extensions)</title>
    <description>Monitor Java Virtual Machines via their exposed JMX MBean attributes, operations and notifications</description>
    <use_external_validation>true</use_external_validation>
    <streaming_mode>xml</streaming_mode>
    <use_single_instance>true</use_single_instance>
    <endpoint>
        <args>
            <arg name="name">
                <title>JMX Input Name</title>
                <description>Name of the JMX input</description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>true</required_on_create>
            </arg>
            <arg name="config_file">
                <title>JMX Config File</title>
                <description>Name of the config file.Defaults to config.xml</description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="config_file_dir">
                <title>JMX Config File Directory</title>
                <description>Alternative location for the config files relative to SPLUNK_HOME ie: etc/apps/foobar</description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
            <arg name="polling_frequency">
                <title>Polling Frequency</title>
                <description>How frequently to execute the polling in seconds.Defaults to 60</description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>false</required_on_create>
            </arg>
        </args>
    </endpoint>
</scheme>

Any idea or help would be very welcome.

Thank you and regards,
Guenter

[jmallorquin · Accepted solution]

Hi,

Did you install Java?

Hope I help you

[glentes]

yes, we are using JDK 1.8

[glentes]

found the reason for the problem:
looks like Splunk_TA_jmx expect java to be located in /usr/bin
setting a symbolic link to the java executable solves the problem