Getting Data In
Highlighted

Issues with Splunk_TA_nix and AIX

Contributor

Customer reports various issues with SplunkTAnix with regards to the vmstat.sh, bandwidth.sh, passwd.sh and df.sh functions. They are reporting errors similar to

bandwith.sh
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtrSplunkTAaix/bin/bandwidth.sh" awk: The statement cannot be correctly parsed.
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr
SplunkTAaix/bin/bandwidth.sh" BEGIN {print "Name rxPacketsPS txPacketsPS rxKBPS txKBPS"} ($0 ~ "Name|sar| lo") {next} {Name=$1; rxPacketsPS=$5; txPacketsPS=$7; >>> rxKBPS=? <<<
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr
SplunkTAaix/bin/bandwidth.sh" [-cCgimMnPrsuvZ] [-f addressfamily] [ [-p proto] | [-@ [wparname]] ]
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr
SplunkTAaix/bin/bandwidth.sh" [-D]
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtrSplunkTAaix/bin/bandwidth.sh" netstat: illegal option -- ;
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr
SplunkTAaix/bin/bandwidth.sh" [-n] [-I interface] [interval]
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtrSplunkTAaix/bin/bandwidth.sh" syntax error The source line is 1.
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr
SplunkTAaix/bin/bandwidth.sh" The error context is
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtrSplunkTAaix/bin/bandwidth.sh" The source line is 1.
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr
SplunkTAaix/bin/bandwidth.sh" usage: netstat [-Aaon] [-f address_family] [-@ [wparname]]

passwd.sh
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtrSplunkTAaix/bin/passwd.sh" Could not load module libpthreads.a(shrxpg564.o).
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr
SplunkTAaix/bin/passwd.sh" Could not load program /opt/splunkforwarder/bin/openssl:
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtrSplunkTAaix/bin/passwd.sh" Dependent module libpthreads.a(shrxpg564.o) could not be loaded.
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr
SplunkTAaix/bin/passwd.sh" System error: No such file or directory

vmstat.sh
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtrSplunkTAaix/bin/vmstat.sh" awk: The field -11 cannot be less than 0.
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr
SplunkTAaix/bin/vmstat.sh" The input line number is 5.
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtrSplunkTA_aix/bin/vmstat.sh" The source line number is 1.

df.sh
Filesystem Type is reported as "?" with long filesystem names
eg
ilesystem Type Size Used Avail UsePct MountedOn
/dev/hd4 jfs2 5G 327.734M 4.67995G 7% /
/dev/hd2 jfs2 8G 1.74594G 6.25406G 22% /usr
/dev/hd9var jfs2 4G 482.641M 3.52867G 12% /var
/dev/hd3 jfs2 2G 84.9609M 1.91703G 5% /tmp
/dev/hd1 jfs2 50G 18.7106G 31.2894G 38% /home
/dev/hd11admin jfs2 128M 0.371094M 127.629M 1% /admin
/dev/hd10opt jfs2 2G 372.668M 1.63607G 19% /opt
/dev/livedump ? 256M 0.359375M 255.641M 1% /var/adm/ras/livedump

Tags (3)
0 Karma
Highlighted

Re: Issues with Splunk_TA_nix and AIX

Contributor

Unfortunately there currently are a few issues with running SplunkTA-nix on AIX. This applies to all versions up to and including SplunkTA_nix 5.2.4. However there are a few simple workarounds to resolve this issue.

bandwidth.sh
Replace the AIX stanza with

elif [ "x$KERNEL" = "xAIX" ] ; then
    # Sample output: http://www-01.ibm.com/support/knowledgecenter/ssw_aix_61/com.ibm.aix.performance/nestat_in.htm
    sudo netstat -i -Z; sleep 1;
    CMD='netstat -in'
    FILTER='($0 ~ "Name|sar| lo") {next}'
    FORMAT='{Name=$1; rxPackets_PS=$5; txPackets_PS=$7; rxKB_PS="?"; txKB_PS="?"}'

passwd.sh
Replace line 23 in passwd.sh
From:

CMD='eval date ; eval LIBPATH=$SPLUNK_HOME/lib $SPLUNK_HOME/bin/openssl sha1 $PASSWD_FILE ; cat $PASSWD_FILE'

To:

CMD='eval date ; eval LIBPATH=$SPLUNK_HOME/lib "$SPLUNK_HOME/bin/splunk cmd openssl" sha1 $PASSWD_FILE ; cat $PASSWD_FILE'

vmstat.sh
Replace the AIX stanza with

elif [ "x$KERNEL" = "xAIX" ] ; then
        assertHaveCommand uptime
        assertHaveCommand ps
        assertHaveCommand vmstat
        assertHaveCommandGivenPath /usr/bin/sudo /usr/sbin/swap
        assertHaveCommandGivenPath /usr/bin/svmon
        CMD='eval uptime ; ps -e | wc -l ; ps -em | wc -l ; /usr/bin/sudo /usr/sbin/swap -s ; vmstat 1 1 ; vmstat -s ; svmon;dirname $0/hardware.sh;'
        PARSE_0='NR==1 {loadAvg1mi=0+$(NF-2)} NR==2 {processes=$1} NR==3 {threads=$1-processes }'
        # ps -em inclundes processes with there threads ( at least one), so processes must be excluded to count threads #
        PARSE_1='(NR==4) {swapUsed=0+$(NF-5); swapFree=0+$(NF-1)} (NR==10) {pgPageIn_PS=0+$(NF-13); pgPageOut_PS=0+$(NF-12)}'
        PARSE_2='/^memory / {memTotalMB=$2 / 256 ; memFreeMB=$4 / 256}'
        PARSE_3='/paging space page outs$/ {pgPageOut=$1 ; pgSwapOut="?" }'
        # no pgSwapOut parameter and can't be monitored in AIX (by Jacky Ho, Systex)
        PARSE_4='/cpu context switches$/ {cSwitches=$1} /device interrupts$/ {interrupts=$1 ; forks="?" }'
        PARSE_5='/^CPU_COUNT/ {cpuCount=$2}'
        MASSAGE="$PARSE_0 $PARSE_1 $PARSE_2 $PARSE_3 $PARSE_4 $PARSE_5 $DERIVE"

df.sh
Replace line 42 in df.sh
From:

CMD='eval /usr/sysv/bin/df -n ; /usr/bin/df -kP'

To:

CMD='eval /usr/sysv/bin/df -n |sed "s/:/ :/" ; /usr/bin/df -kP' 

View solution in original post

Highlighted

Re: Issues with Splunk_TA_nix and AIX

Contributor

Note there are two further issues with regards the SplunkTAnix on AIX that may be worth mentioning

cpu.sh
This command uses output from the operating system command sar -P ALL 11
On Linux systems the sar command is available to "non-root" users. Unfortunately on AIX root privilege is required to run this command

vmstat.sh (again)
This command uses outputs from the following operating system commands
uptime, swap vmstat & svmon. Unfortunately on AIX root privilege is also required to run the swap command

In order to workaround theses issues you will need to consider a solution based on one of the following

  1. Run splunk as "root" on AIX Universal Forwarders.
  2. Grant Splunk "sudo" access to the swap and sar commands, without password (and call the swap and sar commands from sudo in vmstat.sh & cpu.sh.
  3. Use Role Based Access Control (RBAC) (see https://www.ibm.com/developerworks/aix/library/au-aix_rbac/index.html)
0 Karma
Highlighted

Re: Issues with Splunk_TA_nix and AIX

Path Finder

As per your comments i have made 2 changes to bandwidth.sh.

elif [ "x$KERNEL" = "xAIX" ] ; then
    # Sample output: http://www-01.ibm.com/support/knowledgecenter/ssw_aix_61/com.ibm.aix.performance/nestat_in.htm
    netstat -i -Z; sleep 1;
    CMD='netstat -in'
    FILTER='($0 ~ "Name|sar| lo") {next}'
    FORMAT='{Name=$1; rxPackets_PS=$5; txPackets_PS=$7; rxKB_PS="?"; txKB_PS="?"}'
elif

separated netstat 2 separate lines and added " around ?.

we could see ? as data in Splunk for rxKBPS, txKBPS. Do you know how can we populate the data for this ?

0 Karma
Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.