All Apps and Add-ons

Why am I getting this error in splunkd.log? "Error executing modular input : Connection timed out"

New Member

I've installed the "Monitoring of Java Virtual Machines with JMX" app on my universal forwarder box (Fedora 26, OpenJDK 1.8, Python 2.7). I'm getting a "connection timed out" error that I can't figure out. I've successfully install the SplunkTAUnix app, so I think my forwarder is working. Any help would be appreciated.

my application is running with PID 66850 and I can connect to it with Jconsole

Troubleshooting:
I've verified that port 8089 is listening on 127.0.0.1

[splunk@fedora1]$ netstat -a
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:8089            0.0.0.0:*               LISTEN

This is the error I'm seeing in splunkd.log

08-15-2017 14:30:36.411 -0500 ERROR ExecProcessor - message from "python /opt/splunkforwarder/etc/apps/SPLUNK4JMX/bin/jmx.py" Error executing modular input : Connection timed out (Connection timed out) : java.lang.RuntimeException: Connection timed out (Connection timed out)
08-15-2017 14:30:36.411 -0500 ERROR ExecProcessor - message from "python /opt/splunkforwarder/etc/apps/SPLUNK4JMX/bin/jmx.py"   at com.splunk.HttpService.send(HttpService.java:370)
08-15-2017 14:30:36.415 -0500 ERROR ExecProcessor - message from "python /opt/splunkforwarder/etc/apps/SPLUNK4JMX/bin/jmx.py"   at com.splunk.Service.send(Service.java:1268)
.............

splunk cmd splunkd print-modinput-config jmx

    <?xml version="1.0" encoding="UTF-8"?>
    <input>
      <server_host>fedora1</server_host>
      <server_uri>https://127.0.0.1:8089</server_uri>
      <session_key>bP7pl5adKsfxtVR9Fm67EsrA3CJMdnIJbpDflNpYX50zuvc6M4XljlG9y4rwWX^vzoHtVEqgVoRwQydOvFayndEauEM^AYOXh7vEuPC1h_J6zT_eAC</session_key>
      <checkpoint_dir>/opt/splunkforwarder/var/lib/splunk/modinputs/jmx</checkpoint_dir>
      <configuration>
        <stanza name="jmx://helloworld" app="SPLUNK4JMX">
          <param name="config_file">config.xml</param>
          <param name="disabled">0</param>
          <param name="host">fedora1</param>
          <param name="index">jmx</param>
          <param name="polling_frequency">5</param>
          <param name="sourcetype">jmx</param>
        </stanza>
      </configuration>
</input>

Here is my config.xml:

<jmxpoller>
<formatter className="com.dtdsoftware.splunk.formatter.TokenizedMBeanNameQuotesStrippedFormatter" />

<jmxserver pid="66850" jvmDescription="nlrs">

<mbean domain="java.lang" properties="type=Memory">

<attribute name="NonHeapMemoryUsage:used" outputname="nonHeapUsed"/>
<attribute name="NonHeapMemoryUsage:max" outputname="nonHeapMax"/>

</mbean>
</jmxserver>
</jmxpoller>

~/bin/splunk cmd splunkd print-modinput-config jmx | ~/bin/splunk cmd /bin/python ./jmx.py --validate-arguments

ERROR Error parsing XML : unexpected element (uri:"", local:"input"). Expected elements are <{}items>

I'm not sure what this error is or if it's related to my problem?

0 Karma

New Member

I've created a new fedora box and did a fresh install the "Monitoring of Java Virtual Machines with JMX" app on my universal forwarder box. I see a different error now?

-bash-4.4$ ~/bin/splunk cmd splunkd print-modinput-config jmx | ~/bin/splunk cmd /bin/python ./jmx.py

[Fatal Error] :-1:-1: Premature end of file.
ERROR Error executing modular input : HTTP 401 -- <?xml version="1.0" encoding="UTF-8"?>
<response>
  <messages>
    <msg type="WARN">call not properly authenticated</msg>
  </messages>
</response>
 : com.splunk.HttpException: HTTP 401 -- <?xml version="1.0" encoding="UTF-8"?>
<response>
  <messages>
    <msg type="WARN">call not properly authenticated</msg>
  </messages>
</response>

        at com.splunk.HttpException.create(HttpException.java:84)
        at com.splunk.HttpService.send(HttpService.java:399)
        at com.splunk.Service.send(Service.java:1268)
        at com.splunk.HttpService.get(HttpService.java:151)
        at com.splunk.Entity.refresh(Entity.java:381)
        at com.splunk.Entity.refresh(Entity.java:1)
        at com.splunk.Resource.validate(Resource.java:174)
        at com.splunk.Entity.validate(Entity.java:462)
        at com.splunk.Entity.getContent(Entity.java:157)
        at com.splunk.Entity.getString(Entity.java:279)
        at com.splunk.ServiceInfo.getVersion(ServiceInfo.java:155)
        at com.splunk.modinput.ModularInput.runStateCheckerThread(Unknown Source)
        at com.splunk.modinput.ModularInput.init(Unknown Source)
        at com.splunk.modinput.jmx.JMXModularInput.main(Unknown Source)
0 Karma

New Member

Hello,
i have exactly the same error message since few days. SplunkTAjmx suddenly stopped to work (with jdk1.7, Splunk 6.6.1 on redhat6)

0 Karma

New Member

One of inputs was not able to connect, so authentication failed and all jmx add on stopped. It could be improved to avoid full fail in case of 1 inputs over hundred is failing... try to review your jmx authentication or inputs config.

0 Karma