App - Monitoring of Java Virtual Machines

[saas_sirsidynix]

I've been attempting to install this app within the Universal Forwarder on a Solaris server. I've had some success with installing the app on my Splunk indexer, and connecting to a remote JMX port, but wanted to try this method.

After installing and setting up the .xml file, I get this error in splunkd.log when starting the Universal Forwarder. Can anyone advise where I am going wrong ?

Solaris server is running JRE 1.6.0_45.
Splunk version 6.

Exception in thread "Thread-4" java.lang.UnsatisfiedLinkError: sun.tools.attach.LinuxVirtualMachine.isLinuxThreads()Z
at sun.tools.attach.LinuxVirtualMachine.isLinuxThreads(Native Method)
at sun.tools.attach.LinuxVirtualMachine.(LinuxVirtualMachine.java:325)
at sun.tools.attach.LinuxAttachProvider.attachVirtualMachine(LinuxAttachProvider.java:46)
at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:195)
at com.splunk.modinput.jmx.JMXModularInput$ProcessServerThread.getURLForPid(Unknown Source)
at com.splunk.modinput.jmx.JMXModularInput$ProcessServerThread.getJMXServiceURL(Unknown Source)
at com.splunk.modinput.jmx.JMXModularInput$ProcessServerThread.connect(Unknown Source)
at com.splunk.modinput.jmx.JMXModularInput$ProcessServerThread.run(Unknown Source)

Thanks

[Damien_Dallimor]

You might not have libattach.so in your Java environment.

Refer to this from the docs :

JVM "ATTACH" NATIVE LIBRARYS

When connecting directly to local JVM
using a process ID, native librarys
are used. You shouldn't need to do
anything, they are loaded
automatically.

Windows : JRE_HOME/bin/attach.dll
Linux :
JDK_HOME/jre/lib/i386/libattach.so

NOTE : For some reason the Linux
"attach" library is only packaged in
the JRE that is part of a JDK install.
Weird. So if you don't have
libattach.so , you can get it from the
JDK and copy it into the jre lib
directory.

[saas_sirsidynix]

Yep, things seem to be working fine now.

Thanks for your help !

[saas_sirsidynix]

No it wasn't , trying that now.

[Damien_Dallimor]

Is the Splunk process and the JVM process running under the same User ?

[saas_sirsidynix]

Thanks !

The first option didn't work, I got this error :

message from "python jmx.py" Exception in thread "Thread-4" java.lang.NoClassDefFoundError: com/sun/tools/attach/VirtualMachine

The second option has appeared to have done the trick. I get this error now which I need to look into.

04-24-2014 12:18:16.733 +0100 ERROR ExecProcessor - message from "python /opt/splunkforwarder/etc/apps/jmx_ta/bin/jmx.py" host=, jmxServiceURL=, jmxport=0, jvmDescription=XXXX, processID=9999,stanza=jmx://ZZZZ,systemErrorMessage="Unable to open door: target process not responding or HotSpot VM not loaded"

[Damien_Dallimor]

You are going to have to get the JMX Mod Input to use the sun.tools.attach.SolarisVirtualMachine implementation.

A couple of ways to go about this.

Browse to SPLUNK_HOME/etc/apps/jmx_ta/bin/lib/lin

You should see a file called tools-lin.jar

Try 2 things :

1) delete this file

or if that doesn't work

2) replace this file with the tools.jar from the Java Environment on your Solaris host

[saas_sirsidynix]

JAVA_HOME points to /usr/java

I have a libattach.so in /usr/java/jre/lib/i386 and /usr/java/jre/lib/amd64

[saas_sirsidynix]

I read my post again and realised I didn't mention that 🙂

Yes, I'm attempting to connect directly to the PID of a tomcat process.

Thanks