All Apps and Add-ons

Splunk for JMX: Unable to open socket file (local process attachment)

aware74
Explorer

I have installed Splunk For JMX 1.5.7 as an app into the splunk universal forwarder directory on a virtualized CentOS 6.2 server (VMWare 4.1.x), and have configured the app to use local process attachment.

The process ID is being successfully retrieved using a custom script, and I installed the JDK into my home directory on the same target server as the JRE so that I could place libattach.so in $JAVA_HOME/lib/amd64 and, just to be safe, to copy the same exact Java version (Java 6 Update 30) of tools.jar from $HOME/jdk1.6.0_30/lib/tools.jar to $SPLUNK_HOME/etc/apps/SPLUNK4JMX/bin/lib/tools-lin.jar. I also had to remove tools-win.jar from that directory, as it kept trying to use Windows Process Attachment even though I'm on Linux.

Even with these changes, though, I keep getting the following error message in $SPLUNK_HOME/etc/apps/SPLUNK4JMX/logs/splunk4jmx.log:

2012-10-26 21:40:30,621 ERROR [Thread-2] host=, jmxServiceURL=, jmxport=0, jvmDescription=(my description), processID=1267,systemErrorMessage="Unable to open socket file: target process not responding or HotSpot VM not loaded"

So it is probably this line within com.dtdsoftware.splunk.ProcessServerThread.getURLForPid() that is causing this error:

// attach to the target application
final VirtualMachine vm = VirtualMachine.attach(String.valueOf(pid));

I actually get a similar error when I attempt to use the jstack utility installed in my home directory with the JDK:

./jstack -l 1267
1267: Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding

But when I use the -F option:

./jstack -F -l 1267
Attaching to process ID 1267, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 20.5-b03
...

So what is the equivalent for the "-F" option to VirtualMachine.attach() in Java code?

I need to get this to work for critical Java process monitoring.

I can't use RMI from the Splunk server to the Java server because we have firewalls in place and RMI uses random ports every time. I did try to specify the ports using the full service URL (service:jmx:rmi://(ip4):(port1)/jndi/rmi://(ip4):(port2)), but it did not make a difference, even with those ports open on the firewall (random ports were still being used and blocked).

I don't think I can use SOAP or another HTTP-based protocol that may use more dedicated ports because I don't necessarily have an HTTP or SOAP server running on every Java server (but let me know if this is possible in some other way).

This may be a possibly related thread to the issue I'm experiencing with VirtualMachine.attach().

0 Karma
1 Solution

Damien_Dallimor
Ultra Champion

Might be a file permissions issue (on the socket file) ??

Can you try running the poll_jmx.sh command directly as the same user that the target JVM is running under ?

"./poll_jmx.sh config.xml"

Alternatively , you can actually use RMI from the local Universal Forwarder to the target JVM and just set "localhost" in the jmxserver host field in config.xml. As it is all localhost comms, there are no firewall issues with random RMI ports.

View solution in original post

Damien_Dallimor
Ultra Champion

Might be a file permissions issue (on the socket file) ??

Can you try running the poll_jmx.sh command directly as the same user that the target JVM is running under ?

"./poll_jmx.sh config.xml"

Alternatively , you can actually use RMI from the local Universal Forwarder to the target JVM and just set "localhost" in the jmxserver host field in config.xml. As it is all localhost comms, there are no firewall issues with random RMI ports.

Damien_Dallimor
Ultra Champion

Great , its good to have lots of connectivity options 🙂

0 Karma

aware74
Explorer

I wound up going with local RMI connections from the universal forwarder to the target JVM. Less permission issues to worry about, and avoid concerns about somehow destabilizing the running JVM process.

0 Karma

aware74
Explorer

Thanks. I corrected permissions on the JDK that I was using, and that appeared to resolve the above issue.

I updated the command in inputs.conf to be:

[script://sudo -u (java_process_owner) /opt/splunkforwarder/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml]

When I run the above command from the command line as root, it works perfectly.

When root via Splunk runs the script inside of inputs.conf, I instead receive the error:

well-known file is not secure

Thanks,
Mike

0 Karma

aware74
Explorer

In case it is important, the Java process is being run as a non-root user and the splunk forwarder is being run as root (not my configuration, just what it is).

0 Karma

aware74
Explorer

FYI, updating poll_jmx.sh to use $JDK_HOME/bin/java instead of $JAVA_HOME/bin/java (the JRE) produced the same exact error. Maybe I would get a different result if I could move tools_lin.jar out of the way, but if I do, I get the following error message in splunkd.log on the Java server running the universal forwarder:

10-26-2012 22:43:00.820 +0000 ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" Exception in thread "Thread-2" java.lang.NoClassDefFoundError: com/sun/tools/attach/VirtualMachine
0 Karma
*NEW* Splunk Love Promo!
Snag a $25 Visa Gift Card for Giving Your Review!

It's another Splunk Love Special! For a limited time, you can review one of our select Splunk products through Gartner Peer Insights and receive a $25 Visa gift card!

Review:





Or Learn More in Our Blog >>