Splunk Add-on for Java Management Extension: ERROR - Failed to retrieve RMIServer stub

[choehser]

Even after disabling the Firewalls on the Splunk Server, and on the JMX server (WebSphere Application Server), I get these errors:

2015-08-14 15:48:33,810 - com.splunk.modinput.ModularInput -610084 [Thread-7] ERROR  - Failed to retrieve RMIServer stub: javax.naming.NamingException: Error during resolve [Root exception is org.omg.CORBA.TRANSIENT: initial and forwarded IOR inaccessible  vmcid: IBM  minor code: E07  completed: No]
2015-08-14 15:48:33,852 - org.exolab.castor.mapping.Mapping -610126 [Thread-7] INFO   - Loading mapping descriptors from jar:file:/opt/splunk/etc/apps/Splunk_TA_jmx/bin/lib/jmxmodinput.jar!/mapping.xml
2015-08-14 15:48:33,957 - com.splunk.modinput.ModularInput -610231 [Thread-7] INFO   - 1 servers found in stanza jmx://_Splunk_TA_jmx_:Splunk_TA_jmx:PerformanceMonitoring

Any help is welcome!

Regards Chris

[jenshaizmann]

Hello,

I found a solution to get the addon working on a splunk indexer with WAS jmx connection and enabled administrative security.

1. Download IBM SDK
download version 7 from http://www.ibm.com/developerworks/java/jdk/eclipse

2. Set environment variables

JAVA_HOME=D:\ibm_sdk70
SPLUNK_HOME=D:\Splunk

3. copy WAS' trust store to splunk instance
copy trust store to "D:/Splunk/etc/apps/Splunk_TA_jmx/trust.p12"

4. Enhance addon's jmx.py

# Set to True to use the MX4J JMX implementation
# USE_MX4J = True

# Set to True to test SSL
#TEST_SSL = True
TEST_SSL = False

[...]

#BOOTPATH = build_classpath(sep.join([MODINPUT_HOME, 'bin', 'lib', 'boot']), psep)
#if USE_MX4J:
#    BOOTPATH = BOOTPATH + build_classpath(sep.join([MODINPUT_HOME, 'bin', 'lib', 'mx4j_boot']), psep)
#"-Xbootclasspath/p:" + BOOTPATH,

JAVA_ARGS = [JAVA_EXECUTABLE, "-classpath", CLASSPATH,
    #"-Dcom.ibm.CORBA.enableClientCallbacks=true", # unknown property was set while testing
    "-Dcom.ibm.CORBA.ConfigURL=file:D:\\Splunk\\etc\\apps\\Splunk_TA_jmx\\sas.client.props", # required for a secure connection to WAS
    "-Dcom.ibm.SSL.ConfigURL=file:D:\\Splunk\\etc\\apps\\Splunk_TA_jmx\\ssl.client.props", # required for a secure connection to WAS
    #"-Dcom.ibm.CORBA.Debug=true", # enable this to debug
    #"-Dcom.ibm.CORBA.CommTrace=true", # enable this to debug
    #"-Dcom.ibm.CORBA.Debug.Output=D\\Splunk\\etc\\apps\\Splunk_TA_jmx\\corba.log", # enable this to debug
    "-Xms" + MIN_HEAP,
    "-Xmx" + MAX_HEAP, "-Dconfighome=" + CONFIG_HOME,
    "-Dsplunkhome=" + SPLUNK_HOME, JAVA_MAIN_CLASS]
if TEST_SSL:
    TEST_SSL_ARGS = "-Djavax.net.ssl.trustStore=" + SPLUNK_HOME + "/etc/apps/Splunk_TA_jmx/bin/mx4j.ks"
    JAVA_ARGS.insert(-1, TEST_SSL_ARGS)

5. copy sas.client.props from WAS & edit

com.ibm.CORBA.securityEnabled=true

[...]

com.ibm.CORBA.authenticationTarget=BasicAuth
com.ibm.CORBA.authenticationRetryEnabled=true
com.ibm.CORBA.authenticationRetryCount=3
com.ibm.CORBA.validateBasicAuth=false
com.ibm.CORBA.securityServerHost=
com.ibm.CORBA.securityServerPort=
com.ibm.CORBA.loginTimeout=300
com.ibm.CORBA.loginSource=properties

# RMI/IIOP user identity
com.ibm.CORBA.loginUserid=<admin user>
com.ibm.CORBA.loginPassword=<password>

[...]

# Does this client support stateful sessions?
com.ibm.CSI.performStateful=true

# Does this client support/require BasicAuth (userid/password) client authentication?
com.ibm.CSI.performClientAuthenticationRequired=false
com.ibm.CSI.performClientAuthenticationSupported=true

# Does this client support/require SSL client authentication? 
com.ibm.CSI.performTLClientAuthenticationRequired=false
com.ibm.CSI.performTLClientAuthenticationSupported=false

# Note: You can perform BasicAuth (uid/pw) and SSL client authentication (certificate)
# simultaneously, however, the BasicAuth identity will always take precedence at the server.

# Does this client support/require SSL connections?
com.ibm.CSI.performTransportAssocSSLTLSRequired=true
com.ibm.CSI.performTransportAssocSSLTLSSupported=true

# Does this client support/require 40-bit cipher suites when using SSL?
com.ibm.CSI.performMessageIntegrityRequired=false
com.ibm.CSI.performMessageIntegritySupported=true

# Note: This property is only valid when SSL connections are supported or required.
# Does this client support/require 128-bit cipher suites when using SSL?
com.ibm.CSI.performMessageConfidentialityRequired=false
com.ibm.CSI.performMessageConfidentialitySupported=true

6. copy ssl.client.props from WAS & edit

# TrustStore information
com.ibm.ssl.trustStoreName=ClientDefaultTrustStore
com.ibm.ssl.trustStore=D:/Splunk/etc/apps/Splunk_TA_jmx/trust.p12
com.ibm.ssl.trustStorePassword=<password>
com.ibm.ssl.trustStoreType=PKCS12
com.ibm.ssl.trustStoreProvider=IBMJCE
com.ibm.ssl.trustStoreFileBased=true
com.ibm.ssl.trustStoreReadOnly=false

7. copy some WAS libs to addon
Copy following files from D:/IBM/WebSphere/AppServer/runtimes (WAS instance) to D:/Splunk/etc/apps/Splunk_TA_jmx/bin/lib (splunk indexer)

com.ibm.ws.admin.client_8.5.0.jar
com.ibm.ws.ejb.thinclient_8.5.0.jar
com.ibm.ws.orb_8.5.0.jar

Regards, Jens

[vu_le]

Hi,

Just wondering if any one has found a solution to his?

Can't seem to move past this.

[jcoates_splunk]

I'm wondering if you've followed the WAS directions (http://docs.splunk.com/Documentation/AddOns/latest/IBMWAS/About ), which call for an IIOP connection rather than an RMI connection?

[choehser]

I tried to follow the "guide", but not sure if I understood everything the way it was meant, but I tried to...

The URL used, is
service:jmx:iiop://badportal/jndi/corbaname:iiop:badportal:10034/WsnAdminNameService#JMXConnector
this is about what is written as sample in the guide.

I read this as :
jmx over iiop.

Hope it helps to get over this

Thx, Chris

[jenshaizmann]

Hello,

I've the same problem. Did you have already found any solution for this?

Regards Jens

[choehser]

Hi Jens,

sorry, no helping answer so far 😞
In case I get an answer, or get itz to work somehow, I will let all know, as I thought, that others might stumble at this point as well.

Cheers, Chris