Why am I getting this error in splunkd.log? "Error executing modular input : Connection timed out"

[jorymatson]

I've installed the "Monitoring of Java Virtual Machines with JMX" app on my universal forwarder box (Fedora 26, OpenJDK 1.8, Python 2.7). I'm getting a "connection timed out" error that I can't figure out. I've successfully install the Splunk_TA_Unix app, so I think my forwarder is working. Any help would be appreciated.

my application is running with PID 66850 and I can connect to it with Jconsole

Troubleshooting:
I've verified that port 8089 is listening on 127.0.0.1

[splunk@fedora1]$ netstat -a
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:8089            0.0.0.0:*               LISTEN

This is the error I'm seeing in splunkd.log

08-15-2017 14:30:36.411 -0500 ERROR ExecProcessor - message from "python /opt/splunkforwarder/etc/apps/SPLUNK4JMX/bin/jmx.py" Error executing modular input : Connection timed out (Connection timed out) : java.lang.RuntimeException: Connection timed out (Connection timed out)
08-15-2017 14:30:36.411 -0500 ERROR ExecProcessor - message from "python /opt/splunkforwarder/etc/apps/SPLUNK4JMX/bin/jmx.py"   at com.splunk.HttpService.send(HttpService.java:370)
08-15-2017 14:30:36.415 -0500 ERROR ExecProcessor - message from "python /opt/splunkforwarder/etc/apps/SPLUNK4JMX/bin/jmx.py"   at com.splunk.Service.send(Service.java:1268)
.............

splunk cmd splunkd print-modinput-config jmx

    <?xml version="1.0" encoding="UTF-8"?>
    <input>
      <server_host>fedora1</server_host>
      <server_uri>https://127.0.0.1:8089</server_uri>
      <session_key>bP7pl5adKsfxtVR9Fm67EsrA3CJMdnIJbpDflNpYX50zuvc6M4XljlG9y4rwWX^vzoHtVEqgVoRwQydOvFayndEauEM^AYOXh7vEuPC1h_J6zT_eAC</session_key>
      <checkpoint_dir>/opt/splunkforwarder/var/lib/splunk/modinputs/jmx</checkpoint_dir>
      <configuration>
        <stanza name="jmx://helloworld" app="SPLUNK4JMX">
          <param name="config_file">config.xml</param>
          <param name="disabled">0</param>
          <param name="host">fedora1</param>
          <param name="index">jmx</param>
          <param name="polling_frequency">5</param>
          <param name="sourcetype">jmx</param>
        </stanza>
      </configuration>
</input>

Here is my config.xml:

<jmxpoller>
<formatter className="com.dtdsoftware.splunk.formatter.TokenizedMBeanNameQuotesStrippedFormatter" />

<jmxserver pid="66850" jvmDescription="nlrs">

<mbean domain="java.lang" properties="type=Memory">

<attribute name="NonHeapMemoryUsage:used" outputname="nonHeapUsed"/>
<attribute name="NonHeapMemoryUsage:max" outputname="nonHeapMax"/>

</mbean>
</jmxserver>
</jmxpoller>

~/bin/splunk cmd splunkd print-modinput-config jmx | ~/bin/splunk cmd /bin/python ./jmx.py --validate-arguments

ERROR Error parsing XML : unexpected element (uri:"", local:"input"). Expected elements are <{}items>

I'm not sure what this error is or if it's related to my problem?

[jorymatson]

I've created a new fedora box and did a fresh install the "Monitoring of Java Virtual Machines with JMX" app on my universal forwarder box. I see a different error now?

-bash-4.4$ ~/bin/splunk cmd splunkd print-modinput-config jmx | ~/bin/splunk cmd /bin/python ./jmx.py

[Fatal Error] :-1:-1: Premature end of file.
ERROR Error executing modular input : HTTP 401 -- <?xml version="1.0" encoding="UTF-8"?>
<response>
  <messages>
    <msg type="WARN">call not properly authenticated</msg>
  </messages>
</response>
 : com.splunk.HttpException: HTTP 401 -- <?xml version="1.0" encoding="UTF-8"?>
<response>
  <messages>
    <msg type="WARN">call not properly authenticated</msg>
  </messages>
</response>

        at com.splunk.HttpException.create(HttpException.java:84)
        at com.splunk.HttpService.send(HttpService.java:399)
        at com.splunk.Service.send(Service.java:1268)
        at com.splunk.HttpService.get(HttpService.java:151)
        at com.splunk.Entity.refresh(Entity.java:381)
        at com.splunk.Entity.refresh(Entity.java:1)
        at com.splunk.Resource.validate(Resource.java:174)
        at com.splunk.Entity.validate(Entity.java:462)
        at com.splunk.Entity.getContent(Entity.java:157)
        at com.splunk.Entity.getString(Entity.java:279)
        at com.splunk.ServiceInfo.getVersion(ServiceInfo.java:155)
        at com.splunk.modinput.ModularInput.runStateCheckerThread(Unknown Source)
        at com.splunk.modinput.ModularInput.init(Unknown Source)
        at com.splunk.modinput.jmx.JMXModularInput.main(Unknown Source)

[aurelien_cattez]

Hello,
i have exactly the same error message since few days. Splunk_TA_jmx suddenly stopped to work (with jdk1.7, Splunk 6.6.1 on redhat6)

[aurelien_cattez]

One of inputs was not able to connect, so authentication failed and all jmx add on stopped. It could be improved to avoid full fail in case of 1 inputs over hundred is failing... try to review your jmx authentication or inputs config.