Monitoring of Java Virtual Machines with JMX won't start when Splunk is started as a service?

[Haemikeli]

We have Monitoring of Java Virtual Machines with JMX setup on our Splunk forwarder (linux), and it's running fine when executed "./splunk start" from splunk forwarder bin with below logs.

08-29-2022 09:33:57.733 -0600 INFO SpecFiles - Found external scheme definition for stanza="jmx://" from spec file="/opt/splunkforwarder/etc/apps/SPLUNK4JMX/README/inputs.conf.spec" with parameters="activation_key, config_file, config_file_dir, polling_frequency, additional_jvm_propertys, output_type, hec_port, hec_host, hec_endpoint, hec_poolsize, hec_token, hec_https, hec_batch_mode, hec_max_batch_size_bytes, hec_max_batch_size_events, hec_max_inactive_time_before_batch_flush, log_level"

However,  when I tried to start Splunk agent as a service with sudo service splunk start, everything else started fine, and I'm getting the following error in splunkd.log

08-29-2022 09:46:16.519 -0600 ERROR ModularInputs - Introspecting scheme=jmx: Unable to run "python3.7 /opt/splunkforwarder/etc/apps/SPLUNK4JMX/bin/jmx.py --scheme": child failed to start: No such file or directory

08-29-2022 09:46:16.542 -0600 ERROR ModularInputs - Unable to initialize modular input "jmx" defined in the app "SPLUNK4JMX": Introspecting scheme=jmx: Unable to run "python3.7 /opt/splunkforwarder/etc/apps/SPLUNK4JMX/bin/jmx.py --scheme": child failed to start: No such file or directory.

Anyone can point me in the right direction? I've setup Splunk as a service with sudo ./splunk enable boot-start -user splunkuser

I'm suspecting there is a mismatch in permission between splunkuser (splunk owner) and root, but not sure where I should go to correct that.

[kamlesh_vaghela]

@Haemikeli 

Can you please share which Splunk Technology Add-on OR App you are using?

KV

[Haemikeli]

Sorry I missed the info initially.

We deployed the following add-on on the forwarder level

Monitoring of Java Virtual Machines with JMX

https://splunkbase.splunk.com/app/668/

[kamlesh_vaghela]

@Haemikeli 

Can you please go through the dependencies like Splunk, Java Version, etc?

https://www.baboonbones.com/php/markdown.php?document=jmx/README.md

Dependencies
Splunk 7.0+ Enterprise or Cloud
Java Runtime 1.8+
Supported on all Splunk platforms

I hope this will help you.

Thanks
KV
If any of my replies help you to solve the problem Or gain knowledge, an upvote would be appreciated.

[Haemikeli]

The main issue is the this JMX app was able to start correctly if I start it from splunk agent bin folder with ./splunk start command. It fails if I start it as a service with sudo service splunk start.

I think there is some dependency issue with root user but not 100% sure.

[kamlesh_vaghela]

@Haemikeli 

By looking into error,  Universal Forwarders do not come with Python so they cannot run Python scripts.  You'll need a heavy forwarder for that. That's Y I asked you to check dependencies. 

You can try this in your local instane for more.

08-29-2022 09:46:16.542 -0600 ERROR ModularInputs - Unable to initialize modular input "jmx" defined in the app "SPLUNK4JMX": Introspecting scheme=jmx: Unable to run "python3.7 /opt/splunkforwarder/etc/apps/SPLUNK4JMX/bin/jmx.py --scheme": child failed to start: No such file or directory.

I hope this will help you.

Thanks
KV
If any of my replies help you to solve the problem Or gain knowledge, an upvote would be appreciated.

[Haemikeli]

Yes, Python 3.7 was installed on each one of the VMs. I also indicated JMX app was working properly if I startup Splunk with ./splunk start from Splunk bin folder.

However, it's not working with starting it as a Linux service.

I suspect it's different user permission thing that's causing it to fail, but just want to see if anyone else run into this before.