Splunk Add-on for Java Management Extensions - missing data inputs >> jmx

[rileyken2]

I just installed the add on and got the java set up and I actually have jmx data coming into the main index, but I am not able to see jmx under the settings >> data inputs >> jmx

I would like to have the data going to another index, but can not find out how to do this.

here is the output of my print-modinput-config: 

/opt/splunk/bin/splunk cmd splunkd print-modinput-config jmx
<?xml version="1.0" encoding="UTF-8"?>
<input>
<server_host>SRVP01SPLUNK-01</server_host>
<server_uri>https://127.0.0.1:8089</server_uri>
<session_key>n0Zfn422VQQDkWH_MV^wkRCj3Zy_2yZVD^WYBSx84i69_3g2f^Ylatg_Mb^OOhhY0iodEKMOgZer23LjMRt5vlr5342o8g1uCDeQ73rYU6lRZw^Wfo</session_key>
<checkpoint_dir>/opt/data/splunk/modinputs/jmx</checkpoint_dir>
<configuration>
<stanza name="jmx://_Splunk_TA_jmx_:mirth_poc" app="Splunk_TA_jmx">
<param name="config_file">_Splunk_TA_jmx.Splunk_TA_jmx.mirth_poc.xml</param>
<param name="config_file_dir">etc/apps/Splunk_TA_jmx/local/config</param>
<param name="disabled">0</param>
<param name="host">$decideOnStartup</param>
<param name="index">jmx_mirth</param>
<param name="interval">30</param>
<param name="polling_frequency">60</param>
<param name="python.version">python3</param>
<param name="sourcetype">jmx</param>
<param name="start_by_shell">false</param>
</stanza>
</configuration>
</input>

[kamlesh_vaghela]

@rileyken2 

You can find the inputs in Splunk Add-on for Java Management Extensions App.

Navigate to Splunk Add-on for Java Management Extensions and click on input tab. You will see Create Input Button and list of configured input.

jmx is modular input.  So the inputs you configure using above UI will used this modular input for base configurations & the configured inputs are picked for data collection by jmx.  

To verify just create one input and check the inputs.conf in local folder. The configuration will ask for destination app as well so you need to check respective app in case you change it from default selection. 

I hope this will help you.

Thanks
KV
▄︻̷̿┻̿═━一   ?

If any of my reply helps you to solve the problem Or gain knowledge, an upvote would be appreciated.

[rileyken2]

kamlesh, yes I created an input using the input tab, in the app UI, and there is an inputs.conf

[jmx://_Splunk_TA_jmx_:mirth_poc]
config_file = _Splunk_TA_jmx.Splunk_TA_jmx.mirth_poc.xml
config_file_dir = etc/apps/Splunk_TA_jmx/local/config
disabled = 0
index = jmx_mirth
polling_frequency = 60
sourcetype = jmx

Notice in the inputs.conf file the index=jmx_mirth, this is a new index I created for the jmx data, but the data is being sent to the main index. Any ideas how to get the jmx data to go to the new jmx_mirth index?

jmx_servers.conf

[default]

[test]
destinationapp = Splunk_TA_jmx
host = 10.30.4.6
jmxport = 9999
lookupPath = /jmxrmi
protocol = rmi
stubSource = jndi

jmx_tasks.conf

[default]

[test_input_2]
description = more testing
destinationapp = Splunk_TA_jmx
index = jmx_mirth
interval = 60
servers = Splunk_TA_jmx:test
sourcetype = jmx
templates = Splunk_TA_jmx:Predefined_Get_All_Template
disabled = 0

[rileyken2]

Kamlesh, I am also getting this error in the Messages dropdown in the main UI --> Unable to initialize modular input "jmx" defined in the app "Splunk_TA_jmx": Introspecting scheme=jmx: script running failed (exited with code 127)..

If I run this cmd :

/opt/splunk/bin/splunk cmd python /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py --scheme

I get this output:

<scheme>
<title>JMX (Java Management Extensions)</title>
<description>Monitor Java Virtual Machines via their exposed JMX MBean attributes, operations and notifications</description>
<use_external_validation>true</use_external_validation>
<streaming_mode>xml</streaming_mode>
<use_single_instance>true</use_single_instance>
<endpoint>
<args>
<arg name="name">
<title>JMX Input Name</title>
<description>Name of the JMX input</description>
<data_type>string</data_type>
<required_on_edit>false</required_on_edit>
<required_on_create>true</required_on_create>
</arg>
<arg name="config_file">
<title>JMX Config File</title>
<description>Name of the config file.Defaults to config.xml</description>
<data_type>string</data_type>
<required_on_edit>false</required_on_edit>
<required_on_create>false</required_on_create>
</arg>
<arg name="config_file_dir">
<title>JMX Config File Directory</title>
<description>Alternative location for the config files relative to SPLUNK_HOME ie: etc/apps/foobar</description>
<data_type>string</data_type>
<required_on_edit>false</required_on_edit>
<required_on_create>false</required_on_create>
</arg>
<arg name="polling_frequency">
<title>Polling Frequency</title>
<description>How frequently to execute the polling in seconds.Defaults to 60</description>
<data_type>string</data_type>
<required_on_edit>false</required_on_edit>
<required_on_create>false</required_on_create>
</arg>
</args>
</endpoint>
</scheme>

I am not sure what that error means, or if my scheme is correct..

-ken

[kamlesh_vaghela]

@rileyken2 

Can you please try this cmd ?

splunk cmd splunkd print-modinput-config jmx jmx://YOUR_INPUT_NAME

You can find YOUR_INPUT_NAME in inputs.conf in local folder.

KV 

[rileyken2]

KV,

This I think is my issue when I run this command I do not get a meaningful response

./splunk cmd splunkd print-modinput-config jmx jmx://_Splunk_TA_jmx:test_3

Not sure why my input is not being located?

I also ran the debug:

./splunk cmd splunkd print-modinput-config --debug

I get a jmx output like this:

Found scheme="jmx".

Locating script for scheme="jmx"...

No regular file="/opt/splunk/etc/apps/Splunk_TA_jmx/linux_x86_64/bin/jmx.sh".

No regular file="/opt/splunk/etc/apps/Splunk_TA_jmx/linux_x86_64/bin/jmx.py".

No regular file="/opt/splunk/etc/apps/Splunk_TA_jmx/linux_x86_64/bin/jmx.js".

No regular file="/opt/splunk/etc/apps/Splunk_TA_jmx/linux_x86_64/bin/jmx".

No script found in dir="/opt/splunk/etc/apps/Splunk_TA_jmx/linux_x86_64/bin"

No regular file="/opt/splunk/etc/slave-apps/Splunk_TA_jmx/linux_x86_64/bin/jmx.sh".

No regular file="/opt/splunk/etc/slave-apps/Splunk_TA_jmx/linux_x86_64/bin/jmx.py".

No regular file="/opt/splunk/etc/slave-apps/Splunk_TA_jmx/linux_x86_64/bin/jmx.js".

No regular file="/opt/splunk/etc/slave-apps/Splunk_TA_jmx/linux_x86_64/bin/jmx".

No script found in dir="/opt/splunk/etc/slave-apps/Splunk_TA_jmx/linux_x86_64/bin"

No regular file="/opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.sh".

Found script "/opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" to handle scheme "jmx".

setup_interpretter(): path=/opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py --scheme schemeName=jmx python.version=python3

XML scheme path "/scheme/title": "title" -> "JMX (Java Management Extensions)"

XML scheme path "/scheme/description": "description" -> "Monitor Java Virtual Machines via their exposed JMX MBean attributes, operations and notifications"

XML scheme path "/scheme/use_external_validation": "use_external_validation" -> "true"

XML scheme path "/scheme/streaming_mode": "streaming_mode" -> "xml"

XML scheme path "/scheme/use_single_instance": "use_single_instance" -> "true"

XML arg path  "/scheme/endpoint/args/arg": "name" -> "name"

XML arg path  "/scheme/endpoint/args/arg/title": "title" -> "JMX Input Name"

XML arg path  "/scheme/endpoint/args/arg/description": "description" -> "Name of the JMX input"

XML arg path  "/scheme/endpoint/args/arg/data_type": "data_type" -> "string"

XML arg path  "/scheme/endpoint/args/arg/required_on_edit": "required_on_edit" -> "false"

XML arg path  "/scheme/endpoint/args/arg/required_on_create": "required_on_create" -> "true"

XML arg path  "/scheme/endpoint/args/arg": "name" -> "config_file"

XML arg path  "/scheme/endpoint/args/arg/title": "title" -> "JMX Config File"

XML arg path  "/scheme/endpoint/args/arg/description": "description" -> "Name of the config file.Defaults to config.xml"

XML arg path  "/scheme/endpoint/args/arg/data_type": "data_type" -> "string"

XML arg path  "/scheme/endpoint/args/arg/required_on_edit": "required_on_edit" -> "false"

XML arg path  "/scheme/endpoint/args/arg/required_on_create": "required_on_create" -> "false"

XML arg path  "/scheme/endpoint/args/arg": "name" -> "config_file_dir"

XML arg path  "/scheme/endpoint/args/arg/title": "title" -> "JMX Config File Directory"

XML arg path  "/scheme/endpoint/args/arg/description": "description" -> "Alternative location for the config files relative to SPLUNK_HOME ie: etc/apps/foobar"

XML arg path  "/scheme/endpoint/args/arg/data_type": "data_type" -> "string"

XML arg path  "/scheme/endpoint/args/arg/required_on_edit": "required_on_edit" -> "false"

XML arg path  "/scheme/endpoint/args/arg/required_on_create": "required_on_create" -> "false"

XML arg path  "/scheme/endpoint/args/arg": "name" -> "polling_frequency"

XML arg path  "/scheme/endpoint/args/arg/title": "title" -> "Polling Frequency"

XML arg path  "/scheme/endpoint/args/arg/description": "description" -> "How frequently to execute the polling in seconds.Defaults to 60"

XML arg path  "/scheme/endpoint/args/arg/data_type": "data_type" -> "string"

XML arg path  "/scheme/endpoint/args/arg/required_on_edit": "required_on_edit" -> "false"

XML arg path  "/scheme/endpoint/args/arg/required_on_create": "required_on_create" -> "false"

Setting up values from introspection for scheme "jmx".

Setting "title" to "JMX (Java Management Extensions)".

Setting "description" to "Monitor Java Virtual Machines via their exposed JMX MBean attributes, operations and notifications".

Setting "use_single_instance" to true.

Setting "use_external_validation" to true.

Setting "title" to "JMX Config File".

Setting "description" to "Name of the config file.Defaults to config.xml".

Setting "required_on_create" to false.

Setting "required_on_edit" to false.

Setting "title" to "JMX Config File Directory".

Setting "description" to "Alternative location for the config files relative to SPLUNK_HOME ie: etc/apps/foobar".

Setting "required_on_create" to false.

Setting "required_on_edit" to false.

Setting "title" to "JMX Input Name".

Setting "description" to "Name of the JMX input".

Setting "title" to "Polling Frequency".

Setting "description" to "How frequently to execute the polling in seconds.Defaults to 60".

Setting "required_on_create" to false.

Setting "required_on_edit" to false.

Introspection setup completed for scheme "jmx".

===================================================

[kamlesh_vaghela]

@rileyken2 

./splunk cmd splunkd print-modinput-config jmx jmx://_Splunk_TA_jmx:test_3

This will not work bcoz test_3 is jmx_task. And there will be a single input which will pull all the task and start data collection.

can you please execute this?

./splunk cmd splunkd print-modinput-config jmx

I'm getting some deprecation errors. like.

/Applications/Splunk/bin/splunk  cmd splunkd print-modinput-config jmx
<stderr> Introspecting scheme=alerts_ttl_modular_input:  /Applications/Splunk/lib/python2.7/site-packages/urllib3/contrib/pyopenssl.py:47: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release.
<stderr> Introspecting scheme=alerts_ttl_modular_input:    from cryptography import x509
<stderr> Introspecting scheme=ar_initialization_modular_input:  /Applications/Splunk/lib/python2.7/site-packages/urllib3/contrib/pyopenssl.py:47: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release.
<stderr> Introspecting scheme=ar_initialization_modular_input:    from cryptography import x509
<stderr> Introspecting scheme=cloudgateway_modular_input:  /Applications/Splunk/lib/python2.7/site-packages/urllib3/contrib/pyopenssl.py:47: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release.
<stderr> Introspecting scheme=cloudgateway_modular_input:    from cryptography import x509
<stderr> Introspecting scheme=device_role_modular_input:  /Applications/Splunk/lib/python2.7/site-packages/urllib3/contrib/pyopenssl.py:47: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release.
<stderr> Introspecting scheme=device_role_modular_input:    from cryptography import x509
<stderr> Introspecting scheme=drone_mode_subscription_modular_input:  /Applications/Splunk/lib/python2.7/site-packages/urllib3/contrib/pyopenssl.py:47: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release.
<stderr> Introspecting scheme=drone_mode_subscription_modular_input:    from cryptography import x509
<stderr> Introspecting scheme=ibm_was_jmx:  Exception in thread "main" java.lang.NoClassDefFoundError: javax/xml/bind/JAXBContext

😕

[rileyken2]

Sure here is the output of the cmd:

root@SRVP01SPLUNK-01:/opt/splunk/bin# ./splunk cmd splunkd print-modinput-config jmx
<?xml version="1.0" encoding="UTF-8"?>
<input>
<server_host>SRVP01SPLUNK-01</server_host>
<server_uri>https://127.0.0.1:8089</server_uri>
<session_key>AwOYJfLGkcFJB5oG2SKWDWX4RX5pmuUvn4jiiWBaJZokvj393GZSZyjVOyjGGcXl^e28Wo48KOPoCkR7EWZxIKQYKUP^GulQdZ1nqW^uBb3CXzYHulBk</session_key>
<checkpoint_dir>/opt/data/splunk/modinputs/jmx</checkpoint_dir>
<configuration>
<stanza name="jmx://test_3" app="Splunk_TA_jmx">
<param name="config_file">config.xml</param>
<param name="config_file_dir">etc/apps/Splunk_TA_jmx/local/config</param>
<param name="disabled">0</param>
<param name="host">$decideOnStartup</param>
<param name="index">jmx_mirth</param>
<param name="interval">30</param>
<param name="polling_frequency">60</param>
<param name="python.version">python3</param>
<param name="sourcetype">jmx</param>
<param name="start_by_shell">false</param>
</stanza>
</configuration>
</input>

[kamlesh_vaghela]

cool @rileyken2 

you are getting expected output. Is data collection working proper?

[rileyken2]

I am getting data into the main index (I am not sure why or how this is happening) should be jmx_mirth index

if I add another input/server/template it has no impact

[kamlesh_vaghela]

Can you please confirm the index name in configured input?  May be it is configured default index.

 

 

[rileyken2]

[kamlesh_vaghela]

Well this should work. Any indexing error ?

[rileyken2]

this query: 

index=_internal host="splun_server" source=*web_service.log log_level=ERROR

returned one record in the past 24 hours (from yesterday afternoon)

2021-07-25 14:05:53,683 ERROR [60fda8000b7ff85d06c850] config:146 - [HTTP 401] Client is not authenticated Traceback (most recent call last): File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/config.py", line 144, in getServerZoneInfoNoMem return times.getServerZoneinfo() File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/times.py", line 163, in getServerZoneinfo serverStatus, serverResp = splunk.rest.simpleRequest('/search/timeparser/tz', sessionKey=sessionKey) File "/opt/splunk/lib/python3.7/site-packages/splunk/rest/__init__.py", line 553, in simpleRequest raise splunk.AuthenticationFailed splunk.AuthenticationFailed: [HTTP 401] Client is not authenticated

[rileyken2]

this query shows quite a few errors: 

index=_internal host="SRVP01SPLUNK-01" log_level=ERROR source="/opt/splunk/var/log/splunk/splunkd.log" component=ExecProcessor Splunk_TA_jmx

looks like 12 error per minute, constant..

here they are:

		07-26-2021 09:04:47.827 -0400 ERROR ExecProcessor - message from "/opt/splunk/bin/python2.7 /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" File "src/lxml/parser.pxi", line 601, in lxml.etree._ParserContext._handleParseResultDoc
	7/26/21 9:04:47.827 AM	07-26-2021 09:04:47.827 -0400 ERROR ExecProcessor - message from "/opt/splunk/bin/python2.7 /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" File "src/lxml/parser.pxi", line 1127, in lxml.etree._BaseParser._parseDoc
	7/26/21 9:04:47.827 AM	07-26-2021 09:04:47.827 -0400 ERROR ExecProcessor - message from "/opt/splunk/bin/python2.7 /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" File "src/lxml/etree.pyx", line 3234, in lxml.etree.fromstring
	7/26/21 9:04:47.827 AM	07-26-2021 09:04:47.827 -0400 ERROR ExecProcessor - message from "/opt/splunk/bin/python2.7 /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" token = etree.fromstring(xml_str).find('session_key').text
	7/26/21 9:04:47.827 AM	07-26-2021 09:04:47.827 -0400 ERROR ExecProcessor - message from "/opt/splunk/bin/python2.7 /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" File "/opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py", line 87, in <module>
	7/26/21 9:04:47.827 AM	07-26-2021 09:04:47.827 -0400 ERROR ExecProcessor - message from "/opt/splunk/bin/python2.7 /opt/splunk/etc/apps/Splunk_TA_jmx/bin/jmx.py" Traceback (most recent call last