Splunk add-on for JMX: Failed to retrieve RMIServer stub

[alex3]

Hello Splunk Community!

My team and I have been stuck trying to get the Splunk add-on for JMX working for us. We've installed the add-on to a heavy forwarder and are trying to connect to the local JMX server URL. After doing so, we get the following in our jmx.log:

2021-12-22 17:03:19,883 - com.splunk.modinput.ModularInput -5318 [Thread-2] INFO [] - Failed connection with service:jmx:rmi://hostname/jndi/rmi://hostname:8578/hostname/8577/jmxrmi, connecting with service:jmx:rmi://hostname/jndi/JMXConnector .
2021-12-22 17:03:19,884 - com.splunk.modinput.ModularInput -5319 [Thread-2] ERROR [] - Exception@checkConnector, e=
java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.NoInitialContextException: Need to specify class name in environment or system property, or as an applet parameter, or in an application resource file: java.naming.factory.initial
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369) ~[?:1.8.0_275]
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) ~[?:1.8.0_275]
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:229) ~[?:1.8.0_275]
at com.splunk.jmx.ServerTask.connect(Unknown Source) ~[jmxmodinput.jar:?]
at com.splunk.jmx.ServerTask.checkConnector(Unknown Source) [jmxmodinput.jar:?]
at com.splunk.jmx.Scheduler.run(Unknown Source) [jmxmodinput.jar:?]
Caused by: javax.naming.NoInitialContextException: Need to specify class name in environment or system property, or as an applet parameter, or in an application resource file: java.naming.factory.initial
at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:662) ~[?:1.8.0_275]
at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:313) ~[?:1.8.0_275]
at javax.naming.InitialContext.getURLOrDefaultInitCtx(InitialContext.java:350) ~[?:1.8.0_275]
at javax.naming.InitialContext.lookup(InitialContext.java:417) ~[?:1.8.0_275]
at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1955) ~[?:1.8.0_275]
at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1922) ~[?:1.8.0_275]
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287) ~[?:1.8.0_275]
... 5 more

Our jmx_servers.conf file is configured as follows:

[default]

[jms4]
account_name = username
destinationapp = Splunk_TA_jmx
jmx_url = service:jmx:rmi://hostname/jndi/rmi://hostname:8578/hostname/8577/jmxrmi
protocol = url
account_password = password

We confirmed that the URL works, because we were able to reach it with the following code:

class Scratch {
    public static void main(String[] args) throws Exception {

        final JMXServiceURL jmxUrl = new JMXServiceURL("service:jmx:rmi://hostname/jndi/rmi://hostname:9878/hostname/9877/jmxrmi");
        final Map<String, String[]> props = new HashMap<>();
        props.put("jmx.remote.credentials", new String[]{"username", "password"});
        final JMXConnector jmxConnector = JMXConnectorFactory.connect(jmxUrl, props);
        final MBeanServerConnection mbsc = jmxConnector.getMBeanServerConnection();
        Arrays.asList(mbsc.getDomains()).forEach(System.out::println);
    }

Any assistance is appreciated.

[alex3 · Accepted solution]

We resolved the issue. We needed to specify jmxuser and jmxpass, NOT account_name and account_password in the jmx_servers.conf file. After changing that and a restart of Splunk, it started working.