All Apps and Add-ons

No JMX data?

New Member

My super complicated test .xml with all the comments and JVM stats removed for functional testing purposes:

<jmxpoller>

    <formatter className="com.dtdsoftware.splunk.formatter.TokenizedMBeanNameQuotesStrippedFormatter" />

    <jmxserver host="redacted" jvmDescription="redacted" jmxport="8686" jmxuser="admin" jmxpass="redacted">

        <mbean domain="com.redacted.redacted.web.redacted" properties="name=application-statistics">

            <attribute name="redacted" outputname="redacted" />

        </mbean>
    </jmxserver>
</jmxpoller>

This produces absolutely nothing. No JMX-related output in the logs. I can connect to the JVM with jconsole using the supplied credentials, browse to the configured mbean, jackhammer the Refresh button and get new values. Splunk, however, gets nothing, and leaves no indication as to why. Searching for "host=redacted" or "source='jmx://redacted'" yields nothing.

This is not a complicated setup, nor is it a complicated app. Single server test environment, just trying to get JMX data into Splunk directly instead of having to do some nasty search-time field extraction after indexing data output from JMXTrans text files. It's currently running on JDK 1.7, I tried 1.6 for science to no avail. No idea how to proceed since I'm not getting any errors and there doesn't appear to be a way to raise the log level.

Yes, the data input source is Enabled, and source type is "jmx", the polling interval is 1s.

This is totally making me feel like a dummy. Any assistance would be greatly appreciated.

0 Karma
1 Solution

Ultra Champion

Copying in your answer to me via direct email :

Anyway, I brought up another test
instance of Splunk on a different box
and it appears to be working, so I'm
not sure what the problem is with this
machine. Thanks for your time.

View solution in original post

Ultra Champion

Copying in your answer to me via direct email :

Anyway, I brought up another test
instance of Splunk on a different box
and it appears to be working, so I'm
not sure what the problem is with this
machine. Thanks for your time.

View solution in original post

Ultra Champion

Any errors in the logs ?

Search in Splunk Web : index=_internal ExecProcessor error jmx.py

0 Karma

Path Finder

I've been trying for about a month now to get this working. Extremely frustrating. When you say 8686 is not the only port in a remote JMX handshake, how do we find out what the "secret" ports are?

0 Karma

Ultra Champion

Can you connect remotely using JConsole ?

java.rmi.ConnectException: Connection refused to host is something in your network setup.

Note , 8686 is not the only port in a remote JMX RMI/IIOP handshake.

0 Karma

New Member

Splunk

09-12-2014 12:22:37.757 -0400 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/jmx_ta/bin/jmx.py"    java.net.ConnectException: Connection timed out]"

09-12-2014 12:22:37.757 -0400 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/jmx_ta/bin/jmx.py" host=redacted, jmxServiceURL=, jmxport=8686, jvmDescription=redacted, processID=0,stanza=jmx://redacted,systemErrorMessage="Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: redacted; nested exception is:
0 Karma

Ultra Champion

Can you connect remotely using JConsole ie: run JConsole from the same host where Splunk is running.

0 Karma

New Member

From Glassfish J2EE server domain.xml:

<jmx-connector port="8686" address="0.0.0.0" security-enabled="false" name="system" auth-realm-name="admin-realm" />
<jvm-options>-Dcom.sun.management.jmxremote.local.only=false</jvm-options>

The target server is also monitored via JMX by Zabbix, has JMX endpoints being polled by JMXTrans, and I can use jmxsh to authenticate and browse the mbeans. So, not a problem with other JMXy things.

0 Karma

New Member

Also I can telnet to 8686 on the target server from the Splunk server:

redacted@splunk ~ $ telnet redacted 8686
Trying 10.201.2.67...
Connected to redacted.
Escape character is '^]'.
^]
telnet> quit
Connection closed.
0 Karma

New Member

iptables is off on the target system.

redacted@redacted ~ $ sudo service iptables status
iptables: Firewall is not running.
0 Karma

Ultra Champion

I smell a firewall config most likely.

0 Karma

New Member

Yes!

"Connection refused to host"

These usually get dumped to logs, I wonder why that isn't happening now..

0 Karma