All Apps and Add-ons

Splunk Add-on for Unix and Linux: Why vmstat from the add-on running on a universal forwarder stopped showing numeric data from events, only the headers?

arkadyz1
Builder

We've recently experienced a very strange situation: vmstat from Splunk_TA_nix, running on a Splunk universal forwarder, stopped showing the numbers among the events - just the headers ("memTotalMB memFreeMB memUsedMB"...). We tried looking around in all kinds of conf files, but in the end, restarted the forwarder - and the events started coming again just fine!

Has anyone experienced such a loss of information? Since it's working for us now, I don't need an immediate action, but would like some insight into how it might have happened.

freedomson
Explorer

I had the some issue with:

  • CentOS release 6.2 (Final)
  • Unix addon version 5.2.4

After some long hours I was able to successfully forward data by changing /opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/vmstat.sh

Removed the last lines:

$CMD | tee $TEE_DEST | $AWK "$HEADERIZE $MASSAGE $FILL_BLANKS $PRINTF"  header="$HEADER"
echo "Cmd = [$CMD];  | $AWK '$HEADERIZE $MASSAGE $FILL_BLANKS $PRINTF' header=\"$HEADER\"" >> $TEE_DEST

And replaced them by:

RESULT=$($CMD | tee $TEE_DEST | $AWK "$HEADERIZE $MASSAGE $FILL_BLANKS $PRINTF")
echo $RESULT | awk -F'pgPageOut_PS  ' '{printf "%s\n%s", $1, $2}'

And it worked!

Note:
At the moment from my investigation it seems there is some kind of timeout on script stanzas that we are not able to control in the configuration file of the universal forward.

0 Karma

ravitejaj
Explorer

I have the similar issue .. Did you get this resolution?

0 Karma

ianthompson
New Member

I have a similar problem where we are getting heads and data on different lines , although they are showing the same time stamp.

0 Karma

freedomson
Explorer

I had the some issue with:

  • CentOS release 6.2 (Final)
  • Unix addon version 5.2.4

After some long hours I was able to successfully forward data by changing /opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/vmstat.sh

Removed the last lines:

$CMD | tee $TEE_DEST | $AWK "$HEADERIZE $MASSAGE $FILL_BLANKS $PRINTF"  header="$HEADER"
echo "Cmd = [$CMD];  | $AWK '$HEADERIZE $MASSAGE $FILL_BLANKS $PRINTF' header=\"$HEADER\"" >> $TEE_DEST

And replaced them by:

RESULT=$($CMD | tee $TEE_DEST | $AWK "$HEADERIZE $MASSAGE $FILL_BLANKS $PRINTF")
echo $RESULT | awk -F'pgPageOut_PS  ' '{printf "%s\n%s", $1, $2}'

And it worked!

malmoore
Splunk Employee
Splunk Employee

Did you take a look at any of the log files (both in Splunk and on the system) to see if there was a problem when vmstat.sh was only returning the header?

What version of *nix do you run?

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...