All Apps and Add-ons

Errors on jmx_ta using splunk Universal forwarders

redcorjo
Explorer

Hi,

I am trying to monitor java instances using jmx_ta at Splunk. It works pretty well on splunk server side with full installation, but not at all when using universal forwarders. By reading some documentation, it looks it fails because UF are not with python 2.7. I tried different ways to use deployment manager to deploy a local python 2.7 instance as SLES11 servers are by default 2.6, and also locally at server side with RPMs but any of both with success. What explicit instructions are required to run jmx_ta at one UF Linux instance? This is a very helpful app. I would very much like to have it operational.

Regards,

Jordi

0 Karma
1 Solution

BrendanMcE
Path Finder

redcorjo > The tool complains in the UF about missing directory ${SPLUNK_HOME}/apps/jmx_ta/bin/lib/boot. I workaround it by creating by hand. Is it normal I need to do it by hand or I miss anything?

A fix for this is to create a file on the deployment server
${SPLUNK_HOME}/apps/jmx_ta/bin/lib/boot/ignore

Then edit jmx.py change

def build_classpath(rootdir,sep):

    classpath = ""
    for filename in os.listdir(rootdir):
      classpath = classpath + rootdir+filename+sep
    return classpath

to

def build_classpath(rootdir,sep):

    classpath = ""
    for filename in os.listdir(rootdir):
       if filename != "ignore":
          classpath = classpath + rootdir+filename+sep
    return classpath

This then allows the deployment server to push out the directory that is needed as it is non blank. Then the jmx.py will not complain that the directory can not be found.

View solution in original post

BrendanMcE
Path Finder

redcorjo > The tool complains in the UF about missing directory ${SPLUNK_HOME}/apps/jmx_ta/bin/lib/boot. I workaround it by creating by hand. Is it normal I need to do it by hand or I miss anything?

A fix for this is to create a file on the deployment server
${SPLUNK_HOME}/apps/jmx_ta/bin/lib/boot/ignore

Then edit jmx.py change

def build_classpath(rootdir,sep):

    classpath = ""
    for filename in os.listdir(rootdir):
      classpath = classpath + rootdir+filename+sep
    return classpath

to

def build_classpath(rootdir,sep):

    classpath = ""
    for filename in os.listdir(rootdir):
       if filename != "ignore":
          classpath = classpath + rootdir+filename+sep
    return classpath

This then allows the deployment server to push out the directory that is needed as it is non blank. Then the jmx.py will not complain that the directory can not be found.

BrendanMcE
Path Finder

do you have the details of how you bundled 2.7 in with imx_ta as this would be really useful

0 Karma

ppablo
Retired

(continuation of @redcorjo's comment that was posted as an answer. too many characters in the original post to be converted as one entire comment)

with that, once we use the deploy monitor to distribute the app to all my instances, it is distributed with same python as part of the package. I hope it helps.

Unfortunately, on my end I achieved half of the work, as my JMX_TA works like a charm for port but not for process binding. The tool JMX_TA looks to work on process binding only if Java process to monitor is running on same OS account as Splunk agent. Considering that splunk agents most of the cases is on root acount, it is very rare Java applications to monitor also on root account. I added two messages to Damin Dalimore asking to confirm that point. As a workaround to that issue not yet solved on my end, I implemented an ad-hoc script as part of the tool where the script itself is used to self learn what applications are listening the port 1099 from JMX and it works on discovery mode for monitored JMX instances at all servers from my farm.

At ${SPLUNK_HOME}/etc/deployment-apps/jmx_ta/local/inputs.conf I added this entry

[script://./bin/scanapps.sh]
interval = 21600.0

And script ${SPLUNK_HOME}/etc/deployment-apps/jmx_ta/bin/scanapps.sh

Regards,

Jordi

0 Karma

redcorjo
Explorer

Hi,

First, I compiled python 2.7 in my linux box, and once it was compiled, I installed it using ${SPLUNK_HOME}/etc/deployment-apps/jmx_ta as the target directory to be installed.

./python setup.py install --home ${SPLUNK_HOME}/etc/deployment-apps/jmx_ta

In a second round, I edited ${SPLUNK_HOME}/etc/deployment-apps/jmx_ta/bin/jmx.py including at the first line the hardended path of the new python instance.

Considering I have all my Linux Splunk Universal forwarders based in /usr/splunkforwarder, the first line was something like

!/usr/splunkforwarder/etc/apps/jmx_ta/bin/python

'''
Wrapper Script for the JMX Modular Input

Copyright (C) 2012 Splunk, Inc.
All Rights Reserved

Because Splunk can't directly invoke Java , we use this python wrapper script that
simply proxys through to the Java program
'''

Damien_Dallimor
Ultra Champion

You need to have Python 2.7 installed on the UF and available on the PATH for the user that you are running the UF as.

Once you have done this admin task , there are no different instructions than running on a full Splunk instance.

Are you seeing any errors in splunkd.log ?

Have you enabled the stanza in inputs.conf ?

0 Karma

redcorjo
Explorer

When it is executed the tool to monitor on IP and port it works, but when I try on process it gets always same error

12-12-2014 11:05:37.327 +0000 ERROR ExecProcessor - message from "python /usr/sap/splunkforwarder/etc/apps/jmx_ta/bin/jmx.py" host=, jmxServiceURL=, jmxport=0, jvmDescription=JVM PID 10399, processID=10399,stanza=jmx://java,systemErrorMessage="Unable to open socket file: target process not responding or HotSpot VM not loaded"
12-12-2014 11:05:37.328 +0000 ERROR ExecProcessor - message from "python /usr/sap/splunkforwarder/etc/apps/jmx_ta/bin/jmx.py" host=, jmxServiceURL=, jmxport=0, jvmDescription=JVMBYPID, processID=2333,stanza=jmx://java,systemErrorMessage="Unable to open socket file: target process not responding or HotSpot VM not loaded"

I tried with java 1.6 and 1.7

I tried to jstack with -F from root (same effective user as splunk UF is running) and it works well

0 Karma

redcorjo
Explorer

Hi,

first of all, thank you very much for your attention to my requests. It is very much appreciated.

Thanks to your tips I was able to install a local version of python2.7 and bundle in the same installation as jmx_ta package. This way I can use deploy_monitor to distribute the python 2.7 distribution, and also jmx_ta. I also updated jmx.py to reflect the python version bundled in the package.

By now it is almost working, except two parts.

  1. The tool complains in the UF about missing directory ${SPLUNK_HOME}/apps/jmx_ta/bin/lib/boot. I workaround it by creating by hand. Is it normal I need to do it by hand or I miss anything?
  2. If I use the jmxserver host option with port, it works well, but if I try to use the pid or pidcommand options, it generates a javadump at OS root directory. My splunk engine is running on root account, The java version is 1.6.0

/usr/lib64/jvm/jre/bin/java -version
java version "1.6.0"
Java(TM) SE Runtime Environment (build pxa6460sr15fp1-20140110_01(SR15 FP1))
IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux amd64-64 jvmxa6460sr15-20131231_180656 (JIT enabled, AOT enabled)
J9VM - 20131231_180656
JIT - r9_20130920_46510ifx3
GC - GA24_Java6_SR15_20131231_1152_B180656)
JCL - 20140107_01

0 Karma

redcorjo
Explorer

I tried different approaches, even installing a new python 2.7 from scratch by compiling it at OS level. It was added the scripted header as per your instructions with the new path of python 2.7 but still no success

12-09-2014 18:21:17.508 +0000 INFO SpecFiles - Found external scheme definition for stanza "jmx://" with 3 parameters: config_file, config_file_dir, polling_frequency
12-09-2014 18:21:17.610 +0000 ERROR ModularInputs - Introspecting scheme=jmx: script running failed (exited with code 1).
12-09-2014 18:21:17.610 +0000 ERROR ModularInputs - Unable to initialize modular input "jmx" defined inside the app "jmx_ta": Introspecting scheme=jmx: script running failed (exited with code 1).

It is out of my hands as I am not capable to see how to succeed UF to use 2.7 with all modules and not giving any error. It should be something trivial, but I am not capable to see what is missing.

0 Karma

Damien_Dallimor
Ultra Champion

Run this command as the same user that you are running Splunk as and observe console output :

$SPLUNK_HOME/bin/splunk cmd python $SPLUNK_HOME/etc/apps/jmx_ta/bin/jmx.py --scheme

Substitute $SPLUNK_HOME accordingly based on your install path.

0 Karma

redcorjo
Explorer

I found this note from your link before creating the message at this foro. My question is more specific to Splunk needs. Considering the below 6 items, how can I configure splunk UF to use python2.7 without changing OS root account system wide. I want to keep 2.7 default for splunk only, not for everything at root. I assume there is somewhere to customize python default path on splunk isolated.

  1. have the new custom python 2.7 as /usr/lib64/python2.7 2.I have python 2.6 as default path on /usr/lib64/python2.6 3.Default Pyton at the server is 2.6 4.Splunk UF is running as root 5.splunk UF is installed under base directory /usr/sap/splunkforwarder

The different proposals from http://stackoverflow.com/questions/19256127/two-versions-of-python-on-linux-how-to-make-2-7-the-defa... are referring how to change system wide the default python version, or to apply to one unix account, but they are not granular as I want to isolate only for splunk running process.

Regards,

Jordi

0 Karma

Damien_Dallimor
Ultra Champion

Add a directive at the start of the jmx.py file to wherever your 2.7 executable is.

#!/some/path/mypython
0 Karma

redcorjo
Explorer

Hi,

I have configured the inputs.conf (it already works in the heavyforwarder). This is the error I have in the UF

12-09-2014 12:02:56.784 +0000 ERROR ModularInputs - Unable to initialize modular input "jmx" defined inside the app "jmx_ta": Introspecting scheme=jmx: script running failed (exited with code 1).

To be more explicit on my question. If my server Linux SLES11 where UF is installed...
1 . I have python 2.6 as default path on /usr/lib64/python2.6

  1. I have the new custom python 2.7 as /usr/lib64/python2.7
  2. I have python 2.6 as default path on /usr/lib64/python2.6
  3. Default Pyton at the server is 2.6
  4. Splunk UF is running as root
  5. splunk UF is installed under base directory /usr/sap/splunkforwarder

What explicit configurations I need to add at splunk environment to use python2.7 rather than default phyton2.6? I understand there are some env variables to tune, but I am not clear which ones, and where to apply it for Splunk.

NOTE: I prefer to keep python2.6 as default version at server level. The doubt is how to do it specific to splunk UF processes only.

Thanks in advance!

Jordi

0 Karma

Damien_Dallimor
Ultra Champion

Some useful how to's here : http://stackoverflow.com/questions/19256127/two-versions-of-python-on-linux-how-to-make-2-7-the-defa...

Also , just check that you have a Java Runtime installed on the UF. A missing JRE is a problem I see with folks sometimes.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...