Oracle WebLogic App for Splunk: How to monitor multiple WebLogic admin servers from a single agent?

[sjangampeta]

I installed the Oracle WebLogic App for Splunk on multiple Admin server hosts and am successful with data ingestion to Splunk.
However, I would like to install the app on a Single Admin server host and retrieve stats from multiple WebLogic admin server hosts. Can someone help me in achieving this?

[vmaniku]

This was originally designed for vertical scaling on same host. Still, this might be worth a shot to try... Please update the TA as follows :

In bin/setWlstEnv.sh, add these lines after the existing SERVER1 config:

export ADMIN_HOST_1=localhost
export DOMAIN_PATH_2=[ same path as DOMAIN_PATH_1 ]
export ADMIN_SERVER_2=[SecondAdminServerName]
export ADMIN_HOST_2=[SecondHostName]
export ADMIN_PORT_2=[SecondPort]

Then , in one of the jmx collectors (i.e. bin/wlsCollectJmxDataHourly.py):

Add this line at line 11:

adminsvrhost = os.environ['ADMIN_HOST_'+curr_index]

And change line 12 to:

admin_url = "t3://" + adminsvrhost + ":" + adminsvrport

---

Let us know if this works out.

thanks,
Vasanth

[sjangampeta]

Hi Vasanth, Please find my python script

import os
from java.util import Date
from java.text import SimpleDateFormat
import javax.management
import javax.management.remote

print 'Starting JMX Data Extraction'

curr_index = sys.argv[1]
adminsvr = os.environ['ADMIN_SERVER_'+curr_index]
adminsvrport = os.environ['ADMIN_PORT_'+curr_index]
adminsvrhost = os.environ['ADMIN_HOST_'+curr_index]

admin_url = "t3://" + adminsvrhost + ":" + adminsvrport

this connect method requires the WLST be invoked from the domain directory

connect(url=admin_url,adminServerName=adminsvr)

splunkHomeDir = ''
try:

splunkHomeDir = os.environ['SPLUNK_HOME']
except KeyError:
print "Please set the environment variable SPLUNK_HOME"
sys.exit(1)

logFileName = splunkHomeDir + '\var\log\wls_jmx.log'

logFileName = os.path.join(os.environ['SPLUNK_HOME'], "var", "log", "wls_jmx.log")

logFileName = os.path.join(splunkHomeDir, "var", "log", "wls_jmx.log")

logFileName = os.path.join(splunkHomeDir, "var", "log", "wls_jmx_minute.log")

logFile = open(logFileName,'a')

def writeServerData(dataStr):
currTime=SimpleDateFormat("MMM dd, yyyy hh:mm:ss a zzz").format(Date())
logStr = "timestamp=%s|%s\n" % ( currTime, dataStr )
logFile.write(logStr)

domainRuntime = domainRuntime()

cd('ServerRuntimes')
servers = domainRuntimeService.getServerRuntimes()
if (len(servers) > 0):
for server in servers:
cd ('/ServerRuntimes/' + server.getName() )
serverDataStr = "domain=%s|server=%s|server_state=%s|server_health=%s" % ( domainRuntime.getName(), server.getName(), str(cmo.getState()), str(cmo.getHealthState()) )
writeServerData(serverDataStr)

    cd ('/ServerRuntimes/' + server.getName() + '/JVMRuntime/'  + server.getName())
    jvmDataStr = jvmDataStr + "|heap_size_current=%s" % ( str(cmo.getHeapFreeCurrent()) )
    jvmDataStr = jvmDataStr + "|heap_free_percent=%s" % ( str(cmo.getHeapFreePercent()) )
    writeServerData(jvmDataStr)

    cd ('/ServerRuntimes/' + server.getName() + '/ThreadPoolRuntime/ThreadPoolRuntime')
    threadPoolDataStr = "domain=%s|server=%s|throughput=%s" % ( domainRuntime.getName(), server.getName(), str(cmo.getThroughput()) )
    threadPoolDataStr = threadPoolDataStr + "|total_threads=%s" % ( str(cmo.getExecuteThreadTotalCount()) )
    threadPoolDataStr = threadPoolDataStr + "|completed_reqs=%s" % ( str(cmo.getCompletedRequestCount()) )
    threadPoolDataStr = threadPoolDataStr + "|hogging_threads=%s" % ( str(cmo.getHoggingThreadCount()) )
    writeServerData(threadPoolDataStr)

    jmsRuntime = server.getJMSRuntime();
    jmsServers = jmsRuntime.getJMSServers();

    for jmsServer in jmsServers:
        jmsServerDataStr = "domain=%s|server=%s|jms_server=%s|JMSQueue=%s" % ( domainRuntime.getName(), server.getName(), jmsServer.getName(),str(jmsServer.getDestinations()) )
        jmsServerDataStr = jmsServerDataStr + "|jms_current_messages_count=%s|jms_pending_messages_count=%s|jms_messages_high=%s" % ( str(jmsServer.getMessagesCurrentCount()), str(jmsServer.getMessagesPendingCount()),str(jmsServer.getMessagesHighCount()) )
        jmsServerDataStr = jmsServerDataStr + "|jms_current_connection_count=%s|jms_messages_received=%s" % ( str(jmsRuntime.getConnectionsCurrentCount()),str(jmsServer.getMessagesReceivedCount()))
        writeServerData(jdbcDataStr)

    jdbcRuntime = server.getJDBCServiceRuntime();
    datasources = jdbcRuntime.getJDBCDataSourceRuntimeMBeans();
    for datasource in datasources:
            jdbcDataStr = "domain=%s|server=%s|State=%s" % ( domainRuntime.getName(), server.getName(), datasource.getState() )
            jdbcDataStr = jdbcDataStr + "|jdbc_datasource_name=%s|jdbc_active_connection_count=%s|jdbc_active_connections_high=%s" % ( datasource.getName(), repr(datasource.getActiveConnectionsCurrentCount()), repr(datasource.getActiveConnectionsHighCount()) )
            jdbcDataStr = jdbcDataStr + "|jdbc_connection_waiting_count=%s|jdbc_waiting_for_connections_high=%s|jdbc_waiting_for_connections_total=%s" % ( repr(datasource.getWaitingForConnectionCurrentCount()), repr(datasource.getWaitingForConnectionHighCount()), str(datasource.getWaitingForConnectionTotal()) )
            jdbcDataStr = jdbcDataStr + "|jdbc_available_connections=%s|CurrentCapacity=%s|CurrentCapacityHigh=%s|FailedReserveRequest=%s|FailuresToReconnect=%s|LeakedConnections=%s|NumAvailable=%s|NumUnavailable=%s" % ( repr(datasource.getNumAvailable()), str(datasource.getCurrCapacity()), str(datasource.getCurrCapacityHighCount()), str(datasource.getFailedReserveRequestCount()), str(datasource.getFailuresToReconnectCount()), str(datasource.getLeakedConnectionCount()), str(datasource.getNumAvailable()), str(datasource.getNumUnavailable()) )
            writeServerData(jdbcDataStr)


            userLockRuntime = server.getServerSecurityRuntime().getDefaultRealmRuntime().getUserLockoutManagerRuntime();
            userLockStr = "domain=%s|server=%s" % ( domainRuntime.getName(), server.getName() )
            userLockStr = userLockStr + "|locked_users_count=%s|invalid_logins_total=%s" % ( repr(userLockRuntime.getLockedUsersCurrentCount()) , repr(userLockRuntime.getInvalidLoginAttemptsTotalCount()) )
            writeServerData(userLockStr)

    workmanagerMaxService = server.getMaxThreadsConstraintRuntimes();

    for workmanager in workmanagerMaxService:
        workManagerMaxDataStr = "domain=%s|server=%s|WorkmanagerName=%s|ExecutingRequests=%s|DeferredRequests=%s" % ( domainRuntime.getName(), server.getName(), workmanager.getName(), (str(workmanager.getExecutingRequests())), str(workmanager.getDeferredRequests()))
        writeServerData(workManagerMaxDataStr)

    workmanagerMinService = server.getMinThreadsConstraintRuntimes();   

    for workmanager in workmanagerMinService:
        workManagerMinDataStr = "domain=%s|server=%s|WorkmanagerName=%s|CompletedRequests=%s" % (domainRuntime.getName(), server.getName(), workmanager.getName(), str(workmanager.getCompletedRequests()))
        workManagerMinDataStr = workManagerMinDataStr + "|CurrentWaitTime=%s|ExecutingRequests=%s|MaxWaitTime=%s|MustRunCount=%s" % (str(workmanager.getCurrentWaitTime()),str(workmanager.getExecutingRequests()), str(workmanager.getMaxWaitTime()), str(workmanager.getMustRunCount()) )
        workManagerMinDataStr = workManagerMinDataStr + "|OutOfOrderExecutionCount=%s|PendingRequests=%s" % (str(workmanager.getOutOfOrderExecutionCount()), str(workmanager.getPendingRequests()) )
        writeServerData(workManagerMinDataStr)

Commented out to add code below, which includes domain and server calls.

cd('/AppRuntimeStateRuntime/AppRuntimeStateRuntime')

appList = cmo.getApplicationIds()

for app in appList:

appStateStr = "app=%s|app_state=%s" % ( app , cmo.getIntendedState(app) )

writeServerData( appStateStr );

cd('/AppRuntimeStateRuntime/AppRuntimeStateRuntime')
appList = cmo.getApplicationIds()
for app in appList:
appStateStr = "domain=%s|server=%s|app=%s|app_state=%s" % ( domainRuntime.getName(), server.getName(), app , cmo.getIntendedState(app) )
writeServerData( appStateStr );

[sjangampeta]

Thanks Vasanth for your response. I made changes to TA as you mentioned, now when i'm trying to execute it's throwing below error

I

nitializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

Starting JMX Data Extraction
Connecting to t3://localhost:5309 with userid weblogic ...
Successfully connected to Admin Server 'test_admin' that belongs to domain 'test'.

Warning: An insecure protocol was used to connect to the
server. To ensure on-the-wire security, the SSL port or
Admin port should be used instead.

Location changed to domainRuntime tree. This is a read-only tree with DomainMBean as the root.
For more help, use help(domainRuntime)

Problem invoking WLST - Traceback (innermost last):
  File "/u02/apps/splunkforwarder/etc/apps/Function1_WLS_Admin_nix_TA1/bin/wlsCollectJmxDataMinute.py", line 50, in ?
NameError: jvmDataStr

Could you please let me know, why it's throwing NameError ?

[vmaniku]

Can you post your copy of wlsCollectJmxDataMinute.py. I'm assuming this is the file you updated.