SPLUNK4JMX Not polling data from specified servers

[stcrispan]

I have installed the SPLUNK4JMX app on my Prod box. I had already gotten this app to work on my Splunk Dev box, and I anticipated simply copying the configs, changing the Index names, and off I would go. It did not happen this way.

Instead, under the Data Inputs, the JMX (Java Management Extensions) input method is missing. I've spent the past 4 hours digging around in Splunk Answers trying to troubleshoot.

One particularly useful method suggested ensuring that your $SPLUNK_HOME and $JAVA_HOME environment variables were set correctly. Check.

Another suggested checking to make sure your Java could be called successfully. Check. Running an updated version, too. JRE1.8.0_161. Check.

Someone suggested checking to make sure you weren't running multiple versions (or too new a version) of python. Version 2.7, check.

Someone provided a check for the --scheme file:
$SPLUNK_HOME/bin/splunk cmd $SPLUNK_HOME/bin/python $SPLUNK_HOME/etc/apps/SPLUNK4JMX/bin/jmx.py --scheme

Ok, now we're getting somewhere. On the dev box which works, I get this:

<scheme>
    <title>JMX (Java Management Extensions)</title>
    <description>Monitor Java Virtual Machines via their exposed JMX MBean attributes, operations and notifications</description>
    <use_external_validation>true</use_external_validation>
    <streaming_mode>xml</streaming_mode>
    <use_single_instance>true</use_single_instance>
    <endpoint>
        <args>
            <arg name="name">
                <title>JMX Input Name</title>
                <description>Name of the JMX input</description>
                <data_type>string</data_type>
                <required_on_edit>false</required_on_edit>
                <required_on_create>true</required_on_create>
            </arg>

-=SNIP=-  (long, long arguments omitted)


            </arg>
        </args>
    </endpoint>
</scheme>

But on my Prod box...which is missing the Data Input method...I get this:

Traceback (most recent call last):
  File "/data/splunk/etc/apps/SPLUNK4JMX/bin/jmx.py", line 146, in <module>
    do_scheme()
  File "/data/splunk/etc/apps/SPLUNK4JMX/bin/jmx.py", line 61, in do_scheme
    run_java()
  File "/data/splunk/etc/apps/SPLUNK4JMX/bin/jmx.py", line 124, in run_java
    process = Popen(java_args)
  File "/data/splunk/lib/python2.7/subprocess.py", line 390, in __init__
    errread, errwrite)
  File "/data/splunk/lib/python2.7/subprocess.py", line 1024, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

In an effort to troubleshoot further, I used another command, provided by what seemed to be the most knowledgeable yet:

$SPLUNK_HOME/bin/splunk cmd splunkd print-modinput-config jmx | $SPLUNK_HOME/bin/splunk cmd $SPLUNK_HOME/bin/python $SPLUNK_HOME/etc/apps/SPLUNK4JMX/bin/jmx.py

On my Dev box (the one which works) I got this output:

License should have extension of '.lic' or '.license', ignoring file 5 GB license
[Fatal Error] :-1:-1: Premature end of file.
ERROR Error executing modular input : HTTP 401 -- <?xml version="1.0" encoding="UTF-8"?>
<response>
  <messages>
    <msg type="WARN">call not properly authenticated</msg>
  </messages>
</response>
 : com.splunk.HttpException: HTTP 401 -- <?xml version="1.0" encoding="UTF-8"?>
<response>
  <messages>
    <msg type="WARN">call not properly authenticated</msg>
  </messages>
</response>

        at com.splunk.HttpException.create(HttpException.java:84)
        at com.splunk.HttpService.send(HttpService.java:399)
        at com.splunk.Service.send(Service.java:1268)
        at com.splunk.HttpService.get(HttpService.java:151)
        at com.splunk.Entity.refresh(Entity.java:381)
        at com.splunk.Entity.refresh(Entity.java:1)
        at com.splunk.Resource.validate(Resource.java:174)
        at com.splunk.Entity.validate(Entity.java:462)
        at com.splunk.Entity.getContent(Entity.java:157)
        at com.splunk.Entity.getString(Entity.java:279)
        at com.splunk.ServiceInfo.getVersion(ServiceInfo.java:155)
        at com.splunk.modinput.ModularInput.runStateCheckerThread(Unknown Source)
        at com.splunk.modinput.ModularInput.init(Unknown Source)
        at com.splunk.modinput.jmx.JMXModularInput.main(Unknown Source)

But on my Prod box--the non-working one-- I got the same thing as the --scheme errors above:

Introspecting scheme=jmx: script running failed (exited with code 1).
Unable to initialize modular input "jmx" defined inside the app "SPLUNK4JMX": Introspecting scheme=jmx: script running failed (exited with code 1).
Scheme "jmx" is not initialized.
Traceback (most recent call last):
  File "/data/splunk/etc/apps/SPLUNK4JMX/bin/jmx.py", line 154, in <module>
    do_run()
  File "/data/splunk/etc/apps/SPLUNK4JMX/bin/jmx.py", line 58, in do_run
    run_java()
  File "/data/splunk/etc/apps/SPLUNK4JMX/bin/jmx.py", line 124, in run_java
    process = Popen(java_args)
  File "/data/splunk/lib/python2.7/subprocess.py", line 390, in __init__
    errread, errwrite)
  File "/data/splunk/lib/python2.7/subprocess.py", line 1024, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

So does anyone have any suggestions about how to fix the prod box? I read on the APP page that you had to apply for a license, which I did, and I received same ...but durned if I can figure out where to put it.

[stcrispan]

Update: Fixing the Java environment variable didn't break anything else.

Eventually, what did break it were the developers changing the password on the instance. So now I'm back in the same boat. sigh

[stcrispan]

So the problem, as per almost any one of these SPLUNK4JMX issues seems to point at, was Java.

Not a wrong version, because like I said, we were running a recent version of Java 8. Plus, we had our JAVA_HOME environment variable set. The problem was...we had our JAVA_HOME set to the wrong directory. When it had been set, it had been set to

/usr/java/jrel1.8.0_161/bin 

...which is all well and fine, because that's where the Java executables live, right? Except in the jmx.py file, it was calling for

$JAVA_HOME/bin

...which would tell it to look for the Java executables in

/usr/java/jrel1.8.0_161/bin/bin

...and that's a non-starter. (Literally.)

So how did I figure this out? Using the results of the above tests, I vi'd the jmx.py file and followed the script to where it called for Java. We copied the failing command, and gave the same command from a prompt...and it failed there, too. That's when we checked the path and found that it was adding an additional /bin.

Now, we are going to fix the environment variable. Wish us luck...to me this seems like pulling at a loose thread on a sweater. Either you're going to get rid of a loose thread...or you're going to pull it and the sleeve comes off....

[stcrispan · Accepted solution]

So I wrote to Baboon Bones (the maker of SPLUNK4JMX), asking where the new license (started in version 2.5) should go. His response:

Have you installed the App and browsed to it’s setup page yet ?

As per the docs :

“....You can configure your JMX inputs via Manager->DataInputs->JMX....”

There is a field to paste the key into.

Damien Dallimore

My response to him indicated that I don't have the JMX as an option in my Data Inputs.

I'll keep this post updated as I get updates...

[mstjohn_splunk]

Thanks for keep us updated @stcrispan . Hope you get to the bottom of this soon.