I have installed Splunk For JMX 1.5.7 as an app into the splunk universal forwarder directory on a virtualized CentOS 6.2 server (VMWare 4.1.x), and have configured the app to use local process attachment.
The process ID is being successfully retrieved using a custom script, and I installed the JDK into my home directory on the same target server as the JRE so that I could place libattach.so in $JAVA_HOME/lib/amd64 and, just to be safe, to copy the same exact Java version (Java 6 Update 30) of tools.jar from $HOME/jdk1.6.0_30/lib/tools.jar to $SPLUNK_HOME/etc/apps/SPLUNK4JMX/bin/lib/tools-lin.jar. I also had to remove tools-win.jar from that directory, as it kept trying to use Windows Process Attachment even though I'm on Linux.
Even with these changes, though, I keep getting the following error message in $SPLUNK_HOME/etc/apps/SPLUNK4JMX/logs/splunk4jmx.log:
2012-10-26 21:40:30,621 ERROR [Thread-2] host=, jmxServiceURL=, jmxport=0, jvmDescription=(my description), processID=1267,systemErrorMessage="Unable to open socket file: target process not responding or HotSpot VM not loaded"
So it is probably this line within com.dtdsoftware.splunk.ProcessServerThread.getURLForPid() that is causing this error:
// attach to the target application
final VirtualMachine vm = VirtualMachine.attach(String.valueOf(pid));
I actually get a similar error when I attempt to use the jstack utility installed in my home directory with the JDK:
./jstack -l 1267
1267: Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding
But when I use the -F option:
./jstack -F -l 1267
Attaching to process ID 1267, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 20.5-b03
...
So what is the equivalent for the "-F" option to VirtualMachine.attach() in Java code?
I need to get this to work for critical Java process monitoring.
I can't use RMI from the Splunk server to the Java server because we have firewalls in place and RMI uses random ports every time. I did try to specify the ports using the full service URL (service:jmx:rmi://(ip4):(port1)/jndi/rmi://(ip4):(port2)), but it did not make a difference, even with those ports open on the firewall (random ports were still being used and blocked).
I don't think I can use SOAP or another HTTP-based protocol that may use more dedicated ports because I don't necessarily have an HTTP or SOAP server running on every Java server (but let me know if this is possible in some other way).
This may be a possibly related thread to the issue I'm experiencing with VirtualMachine.attach().
... View more