We were able to setup and run the jmx_ta on the splunk app(web app). It was able to connect and report JMX data from remote servers. But, we want to use that in a distributed system. For that we tried installing the same thing on the splunkForwarder installed on the application nodes. We did same configs as before, but it won't start up and just throw the below error,
01-11-2014 01:29:20.623 +0000 ERROR ModularInputs - Introspecting scheme=jmx: script running failed (exited with code 1).
01-11-2014 01:29:20.623 +0000 ERROR ModularInputs - Unable to initialize modular input "jmx" defined inside the app "jmx_ta": Introspecting scheme=jmx: script running failed (exited with code 1).
We checked JAVA_HOME and SPLUNK_HOME is set fine. Do we need Python set too ? Also is there a guide or specific steps for installing this on the forwarder ?
I've had a few issues with this on our setup similar to yours.
I'd suggest running
./splunk cmd <path to your python> <jmx script> --scheme (to show the introspection scheme as shown http://docs.splunk.com/Documentation/Splunk/6.0/AdvancedDev/ModInputsScripts#Define_a_scheme_for_int...)
That was they way I managed to find the issues with ours. Also if deploying via a deployment server then it copies the app but leaves out one of the directories that is empty which caused us issues.
Thanks for the quick response Damien.
Below are the paths set, we restarted the forwarder. We still get the same error no difference.
declare -x PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/home/ubuntu/downloads/Python-2.7.5:/home/ubuntu/downloads/Python-2.7.5"
declare -x PYTHONPATH="/home/ubuntu/downloads/Python-2.7.5"
What are we missing ?