All Apps and Add-ons

Splunk Add-on for Unix and Linux: Why vmstat from the add-on running on a universal forwarder stopped showing numeric data from events, only the headers?

arkadyz1
Builder

We've recently experienced a very strange situation: vmstat from Splunk_TA_nix, running on a Splunk universal forwarder, stopped showing the numbers among the events - just the headers ("memTotalMB memFreeMB memUsedMB"...). We tried looking around in all kinds of conf files, but in the end, restarted the forwarder - and the events started coming again just fine!

Has anyone experienced such a loss of information? Since it's working for us now, I don't need an immediate action, but would like some insight into how it might have happened.

freedomson
Explorer

I had the some issue with:

  • CentOS release 6.2 (Final)
  • Unix addon version 5.2.4

After some long hours I was able to successfully forward data by changing /opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/vmstat.sh

Removed the last lines:

$CMD | tee $TEE_DEST | $AWK "$HEADERIZE $MASSAGE $FILL_BLANKS $PRINTF"  header="$HEADER"
echo "Cmd = [$CMD];  | $AWK '$HEADERIZE $MASSAGE $FILL_BLANKS $PRINTF' header=\"$HEADER\"" >> $TEE_DEST

And replaced them by:

RESULT=$($CMD | tee $TEE_DEST | $AWK "$HEADERIZE $MASSAGE $FILL_BLANKS $PRINTF")
echo $RESULT | awk -F'pgPageOut_PS  ' '{printf "%s\n%s", $1, $2}'

And it worked!

Note:
At the moment from my investigation it seems there is some kind of timeout on script stanzas that we are not able to control in the configuration file of the universal forward.

0 Karma

ravitejaj
Explorer

I have the similar issue .. Did you get this resolution?

0 Karma

ianthompson
New Member

I have a similar problem where we are getting heads and data on different lines , although they are showing the same time stamp.

0 Karma

freedomson
Explorer

I had the some issue with:

  • CentOS release 6.2 (Final)
  • Unix addon version 5.2.4

After some long hours I was able to successfully forward data by changing /opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/vmstat.sh

Removed the last lines:

$CMD | tee $TEE_DEST | $AWK "$HEADERIZE $MASSAGE $FILL_BLANKS $PRINTF"  header="$HEADER"
echo "Cmd = [$CMD];  | $AWK '$HEADERIZE $MASSAGE $FILL_BLANKS $PRINTF' header=\"$HEADER\"" >> $TEE_DEST

And replaced them by:

RESULT=$($CMD | tee $TEE_DEST | $AWK "$HEADERIZE $MASSAGE $FILL_BLANKS $PRINTF")
echo $RESULT | awk -F'pgPageOut_PS  ' '{printf "%s\n%s", $1, $2}'

And it worked!

malmoore
Splunk Employee
Splunk Employee

Did you take a look at any of the log files (both in Splunk and on the system) to see if there was a problem when vmstat.sh was only returning the header?

What version of *nix do you run?

0 Karma
*NEW* Splunk Love Promo!
Snag a $25 Visa Gift Card for Giving Your Review!

It's another Splunk Love Special! For a limited time, you can review one of our select Splunk products through Gartner Peer Insights and receive a $25 Visa gift card!

Review:





Or Learn More in Our Blog >>