Problem Configuring JMX apps

[fhjfong]

Hello,

Is there any documentation on how to configure the JMX apps please?

https://apps.splunk.com/app/668/

This is the error I'm seeing in splund.log

03-16-2015 16:21:41.693 -0400 ERROR ExecProcessor - message from "python E:\Splunk\etc\apps\SPLUNK4JMX\bin\jmx.py" host=server1.company.com, jmxServiceURL=, jmxport=9826, jvmDescription=server1_appserver01, processID=0,stanza=jmx://server1_appserver01,systemErrorMessage="Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
03-16-2015 16:21:41.693 -0400 ERROR ExecProcessor - message from "python E:\Splunk\etc\apps\SPLUNK4JMX\bin\jmx.py" java.net.SocketTimeoutException: Read timed out]"

If I try to run it manually, this is the error message:

E:\Splunk\bin>splunk cmd python E:\Splunk\etc\apps\SPLUNK4JMX\bin\jmx.py < e:\Sp
lunk\etc\apps\SPLUNK4JMX\bin\config\server1_appserver01.xml
ERROR Error parsing XML : unexpected element (uri:"", local:"jmxpoller"). Expect
ed elements are <{}input>
ERROR Error executing modular input : null : java.lang.NullPointerException
at com.splunk.modinput.ModularInput$SplunkConnectionPoller.(Unknow
n Source)
at com.splunk.modinput.ModularInput.init(Unknown Source)
at com.splunk.modinput.jmx.JMXModularInput.main(Unknown Source)

Thanks.

Cheers,
Jack

[fhjfong]

Hello, this is what I've done.

I wrote a dummy program which prints hello world every 10 seconds in an infinite for loop, and I started it with JMX enabled.

C:\jdk1.8.0_11\bin\java -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false PCMDump

I was able to connect to it with JConsole at localhost:9010.

After verifying JConsole is working, I moved onto Splunk4JMX.

I cloned the sample config.xml, called it pcm.xml, and modified the port number to 9010. However, I'm getting this error message:

E:\Splunk\bin>splunk cmd python E:\Splunk\etc\apps\SPLUNK4JMX\bin\jmx.py < E:\Sp
lunk\etc\apps\SPLUNK4JMX\bin\config\pcm.xml
ERROR Error parsing XML : unexpected element (uri:"", local:"jmxpoller"). Expect
ed elements are <{}input>
ERROR Error executing modular input : null : java.lang.NullPointerException
at com.splunk.modinput.ModularInput$SplunkConnectionPoller.(Unknow
n Source)
at com.splunk.modinput.ModularInput.init(Unknown Source)
at com.splunk.modinput.jmx.JMXModularInput.main(Unknown Source)

Thanks.

Cheers,
Jack

[fhjfong]

I have created another xml file:

Now my splunkd.log is filled with:
03-19-2015 23:10:37.558 -0400 WARN SearchOperator:kv - Missing FORMAT for: transform_name='Target_Account_ID_as_dest_nt_domain'
03-19-2015 23:10:37.558 -0400 WARN SearchOperator:kv - Invalid key-value parser, ignoring it, transform_name='Target_Account_ID_as_dest_nt_domain'

[fhjfong]

03-19-2015 23:30:47.938 -0400 WARN SearchOperator:kv - Missing FORMAT for: transform_name='Target_Account_ID_as_dest_nt_domain'
03-19-2015 23:30:47.938 -0400 WARN SearchOperator:kv - Invalid key-value parser, ignoring it, transform_name='Target_Account_ID_as_dest_nt_domain'
03-19-2015 23:30:47.938 -0400 WARN SearchOperator:kv - Missing FORMAT for: transform_name='Security_ID_as_src_nt_domain'
03-19-2015 23:30:47.938 -0400 WARN SearchOperator:kv - Invalid key-value parser, ignoring it, transform_name='Security_ID_as_src_nt_domain'

[Damien_Dallimor]

Why are you trying to run the jmx.py script directly ? This is not documented.

It is a Modular Input. It runs under the context of the Splunk environment.

[fhjfong]

Thanks for your reply Damien.

I didn't know I shouldn't run jmx.py directly.

In splunkd.log, here is the error message:

03-19-2015 23:05:24.424 -0400 ERROR ExecProcessor - message from "python E:\Splunk\etc\apps\SPLUNK4JMX\bin\jmx.py" The root config object(JMXPoller) failed to initialize, stanza : jmx://PCMDump

Any suggestion is appreciated.

[Damien_Dallimor]

You have most likely created a malformed config xml file.
I would suggest checking that you adhere to config.xsd (you'll find this file in the release) and perhaps using an xml editor to help you create conforming xml.
And as always referring and cross referencing with the numerous example xml config files is always useful too.

[fhjfong]

Hello Damien,

I've extracted the config.xsd from jmxmodinput.jar

I used the online xml validator (http://www.freeformatter.com/xml-validator-xsd.html) and verified my config xml file is fully valid.

I created another config file... now the splunkd log is complaining Missing FORMAT for: transform_name='Security_ID_as_src_nt_domain' and Missing FORMAT for: transform_name='Target_Account_ID_as_dest_nt_domain'

Thanks.

[Damien_Dallimor]

Those warning messages have nothing to do with the jmx app

[schose]

Hi,

yes setting up this JMX was also a mess for me. First you have to activate JMX at your JVM. Have you done this? First you should verify with jconsole application from JDK if you can "see" data before trying to collect it using the jmxpoller.

second create a jmx config within the apps bin directory - you can use one of the templates there and comment out the line.

third activate the data input - verify the python script is starting correctly. I had a lib issue in the past.. use splunk cmd /usr/bin/python jmx.py to verify.

regards,

Andreas

[Damien_Dallimor]

systemErrorMessage="Failed to retrieve
RMIServer stub:
javax.naming.CommunicationException
[Root exception is
java.rmi.ConnectIOException: error
during JRMP connection establishment;
nested exception is:

To what extent have your worked through the troubleshooting steps in the documentation ?
Perhaps you have firewall issues ?