No JMX data?

[wstiern]

My super complicated test .xml with all the comments and JVM stats removed for functional testing purposes:

<jmxpoller>

    <formatter className="com.dtdsoftware.splunk.formatter.TokenizedMBeanNameQuotesStrippedFormatter" />

    <jmxserver host="redacted" jvmDescription="redacted" jmxport="8686" jmxuser="admin" jmxpass="redacted">

        <mbean domain="com.redacted.redacted.web.redacted" properties="name=application-statistics">

            <attribute name="redacted" outputname="redacted" />

        </mbean>
    </jmxserver>
</jmxpoller>

This produces absolutely nothing. No JMX-related output in the logs. I can connect to the JVM with jconsole using the supplied credentials, browse to the configured mbean, jackhammer the Refresh button and get new values. Splunk, however, gets nothing, and leaves no indication as to why. Searching for "host=redacted" or "source='jmx://redacted'" yields nothing.

This is not a complicated setup, nor is it a complicated app. Single server test environment, just trying to get JMX data into Splunk directly instead of having to do some nasty search-time field extraction after indexing data output from JMXTrans text files. It's currently running on JDK 1.7, I tried 1.6 for science to no avail. No idea how to proceed since I'm not getting any errors and there doesn't appear to be a way to raise the log level.

Yes, the data input source is Enabled, and source type is "jmx", the polling interval is 1s.

This is totally making me feel like a dummy. Any assistance would be greatly appreciated.

[Damien_Dallimor · Accepted solution]

Copying in your answer to me via direct email :

Anyway, I brought up another test
instance of Splunk on a different box
and it appears to be working, so I'm
not sure what the problem is with this
machine. Thanks for your time.

[Damien_Dallimor]

Any errors in the logs ?

Search in Splunk Web : index=_internal ExecProcessor error jmx.py

[shandman]

I've been trying for about a month now to get this working. Extremely frustrating. When you say 8686 is not the only port in a remote JMX handshake, how do we find out what the "secret" ports are?

[Damien_Dallimor]

Can you connect remotely using JConsole ?

java.rmi.ConnectException: Connection refused to host is something in your network setup.

Note , 8686 is not the only port in a remote JMX RMI/IIOP handshake.

[wstiern]

Splunk

09-12-2014 12:22:37.757 -0400 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/jmx_ta/bin/jmx.py"    java.net.ConnectException: Connection timed out]"

09-12-2014 12:22:37.757 -0400 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/jmx_ta/bin/jmx.py" host=redacted, jmxServiceURL=, jmxport=8686, jvmDescription=redacted, processID=0,stanza=jmx://redacted,systemErrorMessage="Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: redacted; nested exception is:

[Damien_Dallimor]

Can you connect remotely using JConsole ie: run JConsole from the same host where Splunk is running.

[wstiern]

From Glassfish J2EE server domain.xml:

<jmx-connector port="8686" address="0.0.0.0" security-enabled="false" name="system" auth-realm-name="admin-realm" />
<jvm-options>-Dcom.sun.management.jmxremote.local.only=false</jvm-options>

The target server is also monitored via JMX by Zabbix, has JMX endpoints being polled by JMXTrans, and I can use jmxsh to authenticate and browse the mbeans. So, not a problem with other JMXy things.

[wstiern]

Also I can telnet to 8686 on the target server from the Splunk server:

redacted@splunk ~ $ telnet redacted 8686
Trying 10.201.2.67...
Connected to redacted.
Escape character is '^]'.
^]
telnet> quit
Connection closed.

[wstiern]

iptables is off on the target system.

redacted@redacted ~ $ sudo service iptables status
iptables: Firewall is not running.

[Damien_Dallimor]

I smell a firewall config most likely.

[wstiern]

Yes!

"Connection refused to host"

These usually get dumped to logs, I wonder why that isn't happening now..