How to install the Monitoring of Java Virtual Machines on a Universal Forwarder?

[alvn_sulendra]

Hi All,

We planned to install the SPLUNK4JMX in the universal forwarder so that it runs the app in the local machine of the universal forwarder (UF) and sends the data to the indexer. The reason for this is our JVM is using Java 1.6.0 JDK and we don't want to open the ephemeral port range in firewall for sending the data.

So far we done:
Deployed to UF these folder/files as mention in this Question (https://answers.splunk.com/answers/91051/splunk4jmx-universalforwarder-installation.html
):
SplunkUniversalForwarder/etc/apps/SPLUNK4JMX/bin/*
SplunkUniversalForwarder/etc/apps/SPLUNK4JMX/default/inputs.conf
SplunkUniversalForwarder/etc/apps/SPLUNK4JMX/default/app.conf
SplunkUniversalForwarder/etc/apps/SPLUNK4JMX/logs
SplunkUniversalForwarder/etc/apps/SPLUNK4JMX/local

We set the config.xml to run on host=localhost and rmi port that was set in the JVM.
Currently there isn't any data received in the index, and we are not sure what needs to be put in inputs.conf or if is there are other things that need to be configured for this to work.

UF we use is version 6.4.2

Thanks

https://answers.splunk.com/answers/91051/splunk4jmx-universalforwarder-installation.html

[rachaelcrook89]

I found that I had to set the PID number in the config.xml file instead of using localhost/port. Once I set it to use the pid, jmx data started pouring into the index. I still have not found a way to get a bat file or pid file to work, however...

[Damien_Dallimor]

Perhaps if you share your configuration xml , pid script , any error messages , any pertinent information to help diagnose/troubleshoot etc.... then someone will be able to assist you.

[rachaelcrook89]

Sure, thanks Damien! Here's my config, bat file, input, and the entry from the jmx log.

<jmxpoller>
<formatter className="com.dtdsoftware.splunk.formatter.TokenizedMBeanNameQuotesStrippedFormatter" />

<!-- Connect to a local JVM by PID obtained from the output of a custom command -->
<jmxserver pidCommand="D:\program files\SplunkUniversalForwarder\etc\apps\us_ssloansvc_dev_jmx\local\DES.bat" jvmDescription="Another JVM">

<mbean domain="java.lang" properties="type=Threading">

<!-- simple attributes -->
<attribute name="TotalStartedThreadCount" outputname="started"/>
<attribute name="ThreadCpuTimeSupported" outputname="cpuTimeSupported"/>
<attribute name="ThreadCpuTimeEnabled" outputname="cpuTimeEnabled"/>
<attribute name="ThreadCount" outputname="count"/>
<attribute name="ThreadContentionMonitoringSupported" outputname="cmSupported"/>
<attribute name="ThreadContentionMonitoringEnabled" outputname="cmEnabled"/>
<attribute name="SynchronizerUsageSupported" outputname="synchSupported"/>
<attribute name="PeakThreadCount" outputname="peak"/>
<attribute name="ObjectMonitorUsageSupported" outputname="omuSupported"/>
<attribute name="DaemonThreadCount" outputname="daemonCount"/>
<attribute name="CurrentThreadUserTime" outputname="currentUserTime"/>
<attribute name="CurrentThreadCpuTimeSupported" outputname="cpuTimeSupported"/>
<attribute name="CurrentThreadCpuTime" outputname="currentCPUTime"/>

</mbean>


<mbean domain="java.lang" properties="type=Runtime">

<!-- tabular attributes -->
<attribute name="SystemProperties:user.timezone:value" outputname="timezone"/>
<attribute name="SystemProperties:java.vm.vendor:value" outputname="jvmVendor"/>

<!-- simple attributes -->
<attribute name="VmVersion" outputname="jvmVersion"/>
<attribute name="VmVendor" outputname="jvmVendor"/>
<attribute name="VmName" outputname="jvmName"/>
<attribute name="Uptime" outputname="jvmUptime"/>
<attribute name="StartTime" outputname="startedAt"/>
<attribute name="Name" outputname="jvmProcessName"/>

<!-- simple array attribute -->
<attribute name="InputArguments" outputname="inputArgs"/>

</mbean>


<mbean domain="java.lang" properties="type=OperatingSystem">

<!-- simple attributes -->
<attribute name="Version" outputname="version"/>
<attribute name="TotalSwapSpaceSize" outputname="swapSpace"/>
<attribute name="TotalPhysicalMemorySize" outputname="physMem"/>
<attribute name="SystemLoadAverage" outputname="sysLoad"/>
<attribute name="ProcessCpuTime" outputname="cpuTime"/>
<attribute name="OpenFileDescriptorCount" outputname="openFD"/>
<attribute name="Name" outputname="name"/>
<attribute name="MaxFileDescriptorCount" outputname="maxFD"/>
<attribute name="FreeSwapSpaceSize" outputname="freeSwap"/>
<attribute name="FreePhysicalMemorySize" outputname="freePhysMem"/>
<attribute name="CommittedVirtualMemorySize" outputname="committed"/>
<attribute name="AvailableProcessors" outputname="processors"/>
<attribute name="Arch" outputname="arch"/>


</mbean>

<mbean domain="java.lang" properties="type=Compilation">

<!-- simple attributes -->
<attribute name="TotalCompilationTime" outputname="time"/>

</mbean>

<mbean domain="java.lang" properties="type=ClassLoading">

<!-- simple attributes -->
<attribute name="TotalLoadedClassCount" outputname="total"/>
<attribute name="LoadedClassCount" outputname="current"/>
<attribute name="UnloadedClassCount" outputname="unloaded"/>

</mbean>


<mbean domain="java.lang" properties="type=Memory">

<!-- composite attributes -->
<attribute name="NonHeapMemoryUsage:used" outputname="nonHeapUsed"/>
<attribute name="NonHeapMemoryUsage:max" outputname="nonHeapMax"/>
<attribute name="NonHeapMemoryUsage:init" outputname="nonHeapInit"/>
<attribute name="NonHeapMemoryUsage:committed" outputname="nonHeapCommitted"/>

<attribute name="HeapMemoryUsage:used" outputname="heapUsed"/>
<attribute name="HeapMemoryUsage:max" outputname="heapMax"/>
<attribute name="HeapMemoryUsage:init" outputname="heapInit"/>
<attribute name="HeapMemoryUsage:committed" outputname="heapCommitted"/>

<attribute name="ObjectPendingFinalizationCount" outputname="objectsPending"/>

</mbean>


<mbean domain="java.lang" properties="type=MemoryPool,name=*">

<attribute name="Name" outputname="name"/>
<attribute name="Type" outputname="memoryType"/>
<attribute name="MemoryManagerNames" outputname="memManagerNames"/>

<attribute name="PeakUsage:used" outputname="peakUsageUsed"/>
<attribute name="PeakUsage:max" outputname="peakUsageMax"/>
<attribute name="PeakUsage:committed" outputname="peakUsageCommitted"/>
<attribute name="PeakUsage:init" outputname="peakUsageInit"/>

<!-- composite attributes -->
<attribute name="CollectionUsage:used" outputname="colUsageUsed"/>
<attribute name="CollectionUsage:max" outputname="colUsageMax"/>
<attribute name="CollectionUsage:committed" outputname="colUsageCommitted"/>
<attribute name="CollectionUsage:init" outputname="colUsageInit"/>

<!-- composite attributes -->
<attribute name="Usage:used" outputname="usageUsed"/>
<attribute name="Usage:max" outputname="usageMax"/>
<attribute name="Usage:committed" outputname="usageCommitted"/>
<attribute name="Usage:init" outputname="usageInit"/>


</mbean>


<mbean domain="java.lang" properties="type=GarbageCollector,name=*">

<!-- simple array attributes -->
<attribute name="Name" outputname="gcName"/>
<attribute name="MemoryPoolNames" outputname="memPools"/>

<!-- simple attributes -->
<attribute name="CollectionTime" outputname="colTime"/>
<attribute name="CollectionCount" outputname="colCount"/>

<!-- composite-simple attributes -->
<attribute name="LastGcInfo:startTime" outputname="gCStart"/>
<attribute name="LastGcInfo:endTime" outputname="gCEnd"/>
<attribute name="LastGcInfo:id" outputname="gCID"/>
<attribute name="LastGcInfo:duration" outputname="gCDuration"/>
<attribute name="LastGcInfo:GcThreadCount" outputname="gCThreadCount"/>

<!-- composite-tabular-composite-simple attributes --> 

<!-- 
Detailed GC Info for the various heap and non-heap memory pools.

I've included an example of extracting "Eden Space" & "Survivor Space"  attributes for HOTSPOT JVM with default GC setup, its just a copy paste exercise for the other pools

Perm Gen
Perm Gen [shared-ro]
Perm Gen [shared-rw]
Eden Space
Survivor Space
Tenured Gen

etc...


Based on which GC collector and policy you use and which JVM, these memory pool names may differ.


<attribute name="LastGcInfo:memoryUsageBeforeGc:Eden Space:value:used" outputname="eden_beforeGCUsed"/>
<attribute name="LastGcInfo:memoryUsageBeforeGc:Eden Space:value:max" outputname="eden_beforeGCMax"/>
<attribute name="LastGcInfo:memoryUsageBeforeGc:Eden Space:value:init" outputname="eden_beforeGCInit"/>
<attribute name="LastGcInfo:memoryUsageBeforeGc:Eden Space:value:committed" outputname="eden_beforeGCCommitted"/>

<attribute name="LastGcInfo:memoryUsageAfterGc:Eden Space:value:used" outputname="eden_afterGCUsed"/>
<attribute name="LastGcInfo:memoryUsageAfterGc:Eden Space:value:max" outputname="eden_afterGCMax"/>
<attribute name="LastGcInfo:memoryUsageAfterGc:Eden Space:value:init" outputname="eden_afterGCInit"/>
<attribute name="LastGcInfo:memoryUsageAfterGc:Eden Space:value:committed" outputname="eden_afterGCCommitted"/>

<attribute name="LastGcInfo:memoryUsageBeforeGc:Survivor Space:value:used" outputname="survivor_beforeGCUsed"/>
<attribute name="LastGcInfo:memoryUsageBeforeGc:Survivor Space:value:max" outputname="survivor_beforeGCMax"/>
<attribute name="LastGcInfo:memoryUsageBeforeGc:Survivor Space:value:init" outputname="survivor_beforeGCInit"/>
<attribute name="LastGcInfo:memoryUsageBeforeGc:Survivor Space:value:committed" outputname="survivor_beforeGCCommitted"/>

<attribute name="LastGcInfo:memoryUsageAfterGc:Survivor Space:value:used" outputname="survivor_afterGCUsed"/>
<attribute name="LastGcInfo:memoryUsageAfterGc:Survivor Space:value:max" outputname="survivor_afterGCMax"/>
<attribute name="LastGcInfo:memoryUsageAfterGc:Survivor Space:value:init" outputname="survivor_afterGCInit"/>
<attribute name="LastGcInfo:memoryUsageAfterGc:Survivor Space:value:committed" outputname="survivor_afterGCCommitted"/>
-->

</mbean>

</jmxserver>


</jmxpoller>

Contents of batch file:
wmic process where name="bwagent.exe" get processid | MORE +1

inputs.conf

[jmx://input-DES]
config_file = config-DES.xml
polling_frequency = 60
sourcetype = jmx
index = index_name
disabled = 0
interval = 30
_TCP_ROUTING = tcp_group
crcSalt = crcsalt

Log:

2017-10-04 18:53:06,915 - com.dtdsoftware.splunk.config.JMXServer -608 [Thread-2] ERROR - Error processing pid line : ProcessId . Message : For input string: "ProcessId "
2017-10-04 18:53:06,916 - com.dtdsoftware.splunk.config.JMXServer -609 [Thread-2] ERROR - Error processing pid line : 2632 . Message : For input string: "2632 "
2017-10-04 18:53:06,922 - com.dtdsoftware.splunk.config.JMXServer -615 [Thread-2] ERROR - No valid pid command output
2017-10-04 18:53:06,922 - com.dtdsoftware.splunk.config.JMXServer -615 [Thread-2] ERROR - Error obtaining pid from command D:\program files\SplunkUniversalForwarder\etc\apps\us_ssloansvc_dev_jmx\local\DES.bat
2017-10-04 18:53:06,928 - com.splunk.modinput.ModularInput -621 [Thread-2] INFO - 1 servers found in stanza jmx://Triad-DES
2017-10-04 18:53:07,935 - com.splunk.modinput.ModularInput -1628 [Thread-2] ERROR - Exception@checkConnector, e=
java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.254.184.159; nested exception is:
java.net.ConnectException: Connection refused: connect]
at javax.management.remote.rmi.RMIConnector.connect(Unknown Source)
at javax.management.remote.JMXConnectorFactory.connect(Unknown Source)
at javax.management.remote.JMXConnectorFactory.connect(Unknown Source)
at com.splunk.jmx.ServerTask.connect(Unknown Source)
at com.splunk.jmx.ServerTask.checkConnector(Unknown Source)
at com.splunk.jmx.Scheduler.run(Unknown Source)
Caused by: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.254.184.159; nested exception is:
java.net.ConnectException: Connection refused: connect]
at com.sun.jndi.rmi.registry.RegistryContext.lookup(Unknown Source)
at com.sun.jndi.toolkit.url.GenericURLContext.lookup(Unknown Source)
at javax.naming.InitialContext.lookup(Unknown Source)
at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(Unknown Source)
at javax.management.remote.rmi.RMIConnector.findRMIServer(Unknown Source)
... 6 more
Caused by: java.rmi.ConnectException: Connection refused to host: 10.254.184.159; nested exception is:
java.net.ConnectException: Connection refused: connect
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(Unknown Source)
at sun.rmi.transport.tcp.TCPChannel.createConnection(Unknown Source)
at sun.rmi.transport.tcp.TCPChannel.newConnection(Unknown Source)
at sun.rmi.server.UnicastRef.newCall(Unknown Source)
at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
... 11 more
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.DualStackPlainSocketImpl.connect0(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source)
at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at java.net.Socket.(Unknown Source)
at java.net.Socket.(Unknown Source)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(Unknown Source)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(Unknown Source)
... 16 more

[rachaelcrook89]

Figured out why it wasn't working. I had to modify my bat file to the following.

@echo off
for /f "tokens=2" %%i in ('tasklist ^| find "bwagent.exe"') DO echo %%i

[Damien_Dallimor]

What version of the App are you using ?
You have referenced an answer that was written in 2013, ergo , applys to the version of the App at that point in time.

[alvn_sulendra]

Hi Damien,

the application version we use is version 2.4

Thank you

[Damien_Dallimor]

well , three years later , 2.4 is a completely different build.

updated answer here.