Splunk Add-on for Unix and Linux: Why vmstat from the add-on running on a universal forwarder stopped showing numeric data from events, only the headers?

[arkadyz1]

We've recently experienced a very strange situation: vmstat from Splunk_TA_nix, running on a Splunk universal forwarder, stopped showing the numbers among the events - just the headers ("memTotalMB memFreeMB memUsedMB"...). We tried looking around in all kinds of conf files, but in the end, restarted the forwarder - and the events started coming again just fine!

Has anyone experienced such a loss of information? Since it's working for us now, I don't need an immediate action, but would like some insight into how it might have happened.

[freedomson]

I had the some issue with:

• CentOS release 6.2 (Final)
• Unix addon version 5.2.4

After some long hours I was able to successfully forward data by changing /opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/vmstat.sh

Removed the last lines:

$CMD | tee $TEE_DEST | $AWK "$HEADERIZE $MASSAGE $FILL_BLANKS $PRINTF"  header="$HEADER"
echo "Cmd = [$CMD];  | $AWK '$HEADERIZE $MASSAGE $FILL_BLANKS $PRINTF' header=\"$HEADER\"" >> $TEE_DEST

And replaced them by:

RESULT=$($CMD | tee $TEE_DEST | $AWK "$HEADERIZE $MASSAGE $FILL_BLANKS $PRINTF")
echo $RESULT | awk -F'pgPageOut_PS  ' '{printf "%s\n%s", $1, $2}'

And it worked!

Note:
At the moment from my investigation it seems there is some kind of timeout on script stanzas that we are not able to control in the configuration file of the universal forward.

[ravitejaj]

I have the similar issue .. Did you get this resolution?

[ianthompson]

I have a similar problem where we are getting heads and data on different lines , although they are showing the same time stamp.

[freedomson]

I had the some issue with:

• CentOS release 6.2 (Final)
• Unix addon version 5.2.4

After some long hours I was able to successfully forward data by changing /opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/vmstat.sh

Removed the last lines:

$CMD | tee $TEE_DEST | $AWK "$HEADERIZE $MASSAGE $FILL_BLANKS $PRINTF"  header="$HEADER"
echo "Cmd = [$CMD];  | $AWK '$HEADERIZE $MASSAGE $FILL_BLANKS $PRINTF' header=\"$HEADER\"" >> $TEE_DEST

And replaced them by:

RESULT=$($CMD | tee $TEE_DEST | $AWK "$HEADERIZE $MASSAGE $FILL_BLANKS $PRINTF")
echo $RESULT | awk -F'pgPageOut_PS  ' '{printf "%s\n%s", $1, $2}'

And it worked!

[malmoore]

Did you take a look at any of the log files (both in Splunk and on the system) to see if there was a problem when vmstat.sh was only returning the header?

What version of *nix do you run?