topic Re: Issues with Splunk_TA_nix and AIX in Getting Data In

Issues with Splunk_TA_nix and AIX

dshakespeare_sp — Tue, 29 Sep 2020 15:37:16 GMT

Customer reports various issues with Splunk_TA_nix with regards to the vmstat.sh, bandwidth.sh, passwd.sh and df.sh functions. They are reporting errors similar to

bandwith.sh
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr_Splunk_TA_aix/bin/bandwidth.sh" awk: The statement cannot be correctly parsed.
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr_Splunk_TA_aix/bin/bandwidth.sh" BEGIN {print "Name rxPackets_PS txPackets_PS rxKB_PS txKB_PS"} ($0 ~ "Name|sar| lo") {next} {Name=$1; rxPackets_PS=$5; txPackets_PS=$7; >>> rxKB_PS=? <<<
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr_Splunk_TA_aix/bin/bandwidth.sh" [-cCgimMnPrsuvZ] [-f address_family] [ [-p proto] | [-@ [wparname]] ]
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr_Splunk_TA_aix/bin/bandwidth.sh" [-D]
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr_Splunk_TA_aix/bin/bandwidth.sh" netstat: illegal option -- ;
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr_Splunk_TA_aix/bin/bandwidth.sh" [-n] [-I interface] [interval]
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr_Splunk_TA_aix/bin/bandwidth.sh" syntax error The source line is 1.
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr_Splunk_TA_aix/bin/bandwidth.sh" The error context is
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr_Splunk_TA_aix/bin/bandwidth.sh" The source line is 1.
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr_Splunk_TA_aix/bin/bandwidth.sh" usage: netstat [-Aaon] [-f address_family] [-@ [wparname]]

passwd.sh
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr_Splunk_TA_aix/bin/passwd.sh" Could not load module libpthreads.a(shr_xpg5_64.o).
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr_Splunk_TA_aix/bin/passwd.sh" Could not load program /opt/splunkforwarder/bin/openssl:
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr_Splunk_TA_aix/bin/passwd.sh" Dependent module libpthreads.a(shr_xpg5_64.o) could not be loaded.
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr_Splunk_TA_aix/bin/passwd.sh" System error: No such file or directory

vmstat.sh
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr_Splunk_TA_aix/bin/vmstat.sh" awk: The field -11 cannot be less than 0.
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr_Splunk_TA_aix/bin/vmstat.sh" The input line number is 5.
ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/wtr_Splunk_TA_aix/bin/vmstat.sh" The source line number is 1.

df.sh
Filesystem Type is reported as "?" with long filesystem names
eg
ilesystem Type Size Used Avail UsePct MountedOn
/dev/hd4 jfs2 5G 327.734M 4.67995G 7% /
/dev/hd2 jfs2 8G 1.74594G 6.25406G 22% /usr
/dev/hd9var jfs2 4G 482.641M 3.52867G 12% /var
/dev/hd3 jfs2 2G 84.9609M 1.91703G 5% /tmp
/dev/hd1 jfs2 50G 18.7106G 31.2894G 38% /home
/dev/hd11admin jfs2 128M 0.371094M 127.629M 1% /admin
/dev/hd10opt jfs2 2G 372.668M 1.63607G 19% /opt
/dev/livedump ? 256M 0.359375M 255.641M 1% /var/adm/ras/livedump

Re: Issues with Splunk_TA_nix and AIX

dshakespeare_sp — Tue, 29 Sep 2020 15:37:18 GMT

Unfortunately there currently are a few issues with running Splunk_TA-nix on AIX. This applies to all versions up to and including Splunk_TA_nix 5.2.4. However there are a few simple workarounds to resolve this issue.

bandwidth.sh
Replace the AIX stanza with

elif [ "x$KERNEL" = "xAIX" ] ; then
    # Sample output: http://www-01.ibm.com/support/knowledgecenter/ssw_aix_61/com.ibm.aix.performance/nestat_in.htm
    sudo netstat -i -Z; sleep 1;
    CMD='netstat -in'
    FILTER='($0 ~ "Name|sar| lo") {next}'
    FORMAT='{Name=$1; rxPackets_PS=$5; txPackets_PS=$7; rxKB_PS="?"; txKB_PS="?"}'

passwd.sh
Replace line 23 in passwd.sh
From:

CMD='eval date ; eval LIBPATH=$SPLUNK_HOME/lib $SPLUNK_HOME/bin/openssl sha1 $PASSWD_FILE ; cat $PASSWD_FILE'

To:

CMD='eval date ; eval LIBPATH=$SPLUNK_HOME/lib "$SPLUNK_HOME/bin/splunk cmd openssl" sha1 $PASSWD_FILE ; cat $PASSWD_FILE'

vmstat.sh
Replace the AIX stanza with

elif [ "x$KERNEL" = "xAIX" ] ; then
        assertHaveCommand uptime
        assertHaveCommand ps
        assertHaveCommand vmstat
        assertHaveCommandGivenPath /usr/bin/sudo /usr/sbin/swap
        assertHaveCommandGivenPath /usr/bin/svmon
        CMD='eval uptime ; ps -e | wc -l ; ps -em | wc -l ; /usr/bin/sudo /usr/sbin/swap -s ; vmstat 1 1 ; vmstat -s ; svmon;dirname $0/hardware.sh;'
        PARSE_0='NR==1 {loadAvg1mi=0+$(NF-2)} NR==2 {processes=$1} NR==3 {threads=$1-processes }'
        # ps -em inclundes processes with there threads ( at least one), so processes must be excluded to count threads #
        PARSE_1='(NR==4) {swapUsed=0+$(NF-5); swapFree=0+$(NF-1)} (NR==10) {pgPageIn_PS=0+$(NF-13); pgPageOut_PS=0+$(NF-12)}'
        PARSE_2='/^memory / {memTotalMB=$2 / 256 ; memFreeMB=$4 / 256}'
        PARSE_3='/paging space page outs$/ {pgPageOut=$1 ; pgSwapOut="?" }'
        # no pgSwapOut parameter and can't be monitored in AIX (by Jacky Ho, Systex)
        PARSE_4='/cpu context switches$/ {cSwitches=$1} /device interrupts$/ {interrupts=$1 ; forks="?" }'
        PARSE_5='/^CPU_COUNT/ {cpuCount=$2}'
        MASSAGE="$PARSE_0 $PARSE_1 $PARSE_2 $PARSE_3 $PARSE_4 $PARSE_5 $DERIVE"

df.sh
Replace line 42 in df.sh
From:

CMD='eval /usr/sysv/bin/df -n ; /usr/bin/df -kP'

To:

CMD='eval /usr/sysv/bin/df -n |sed "s/:/ :/" ; /usr/bin/df -kP'

Re: Issues with Splunk_TA_nix and AIX

dshakespeare_sp — Tue, 29 Sep 2020 17:01:52 GMT

Note there are two further issues with regards the Splunk_TA_nix on AIX that may be worth mentioning

cpu.sh
This command uses output from the operating system command sar -P ALL 11
On Linux systems the sar command is available to "non-root" users. Unfortunately on AIX root privilege is required to run this command

vmstat.sh (again)
This command uses outputs from the following operating system commands
uptime, swap vmstat & svmon. Unfortunately on AIX root privilege is also required to run the swap command

In order to workaround theses issues you will need to consider a solution based on one of the following

Run splunk as "root" on AIX Universal Forwarders.
Grant Splunk "sudo" access to the swap and sar commands, without password (and call the swap and sar commands from sudo in vmstat.sh & cpu.sh.
Use Role Based Access Control (RBAC) (see https://www.ibm.com/developerworks/aix/library/au-aix_rbac/index.html)

Re: Issues with Splunk_TA_nix and AIX

praphulla1 — Tue, 29 Sep 2020 17:01:55 GMT

As per your comments i have made 2 changes to bandwidth.sh.

elif [ "x$KERNEL" = "xAIX" ] ; then
    # Sample output: http://www-01.ibm.com/support/knowledgecenter/ssw_aix_61/com.ibm.aix.performance/nestat_in.htm
    netstat -i -Z; sleep 1;
    CMD='netstat -in'
    FILTER='($0 ~ "Name|sar| lo") {next}'
    FORMAT='{Name=$1; rxPackets_PS=$5; txPackets_PS=$7; rxKB_PS="?"; txKB_PS="?"}'
elif

separated netstat 2 separate lines and added " around ?.

we could see ? as data in Splunk for rxKB_PS, txKB_PS. Do you know how can we populate the data for this ?