Linux Script output is different from _raw

[isaacsu]

Hi all,

I've written script in "/opt/splunk/etc/deployment-apps/testing/bin" named test1.sh which gives me an expected result what i intend the script to be used for.

Script Content.

#!/bin/sh

HEADER='CPU    pctIdle'
HEADERIZE="BEGIN {print \"$HEADER\"}"
PRINTF='END {printf "%-3s  %9.1f\n", CPU, pctIdle}'
AWK=awk
CMD='top -n 1'
PARSE_0='NR==3 {CPU="all"} NR==3 {pctIdle=$8}'

$CMD | $AWK "$HEADERIZE $PARSE_0 $PRINTF"  header="$HEADER"

Script Results

CPU    pctIdle
all       94.9

However, upon running "/opt/splunk/bin/splunk reload deploy-server". The _ raw values of which i receive from my events are all as such.

CPU    pctIdle
           0.0

Can i get some insights on why is this so?

[woodcock]

Generally what happens is that when you run it manually, you are receiving the benefit of the PATH environment variable associated with your user. However, when Splunk runs it as user = splunk, this user has a different value for PATH or maybe no value at all, which means that in your scripts, you should *ALWAYS* fully-qualify the path to every command. So your script would look something like this:

#!/bin/sh
HEADER='CPU    pctIdle'
HEADERIZE="BEGIN {print \"$HEADER\"}"
PRINTF='END {printf "%-3s  %9.1f\n", CPU, pctIdle}'
AWK="/usr/bin/awk"
CMD='/usr/bin/top -n 1'
PARSE_0='NR==3 {CPU="all"} NR==3 {pctIdle=$8}'
$CMD | $AWK "$HEADERIZE $PARSE_0 $PRINTF"  header="$HEADER"

Your coding is very awkwardly convoluted. It should be half as long with no variables at all.

[isaacsu]

Hi Woodcock, based on our observations. We've tried to define both the full path and also the path you've seen in the script above. Unfortunately, both produced the same results. Also, apologies for the convoluted script, i'm still new to this ^_^l

[nickhills]

If you are just interested in CPU data I echo the sentiments below - use the Splunk provided TA as all of this work is done for you.

If, however this is an exercise which you plan to expand further, you can make you Splunk ingestion journey much easier using K=V values as the response from your script. Also make you your life easier by including a timestamp.

CSV output formats make a huge amount of sense if the output from your script is large and repetitive. In those cases you only write the field names (header) out once, and then list all the values, but for a simple output from a script, you will find it much easier if you format your output like so:

06/03/2020 09:36:24 cpu=all pctIdle=94.6
Splunk will automatically break events formatted like this, and the fields will be auto extracted for you

[manjunathmeti]

And where is app/add-on testing is deployed?

[isaacsu]

This is currently being deployed via deployment-apps in my Cluster Master

[manjunathmeti]

Is this script part of scripted input? And how the output of script is forwarded to indexer server?

[isaacsu]

The script i wrote resides in the bin folder of my testing app "/opt/splunk/etc/deployment-apps/testing/bin".

It is being invoked by "/opt/splunk/etc/deployment-apps/testing/local" through the following inputs.conf stanza.

[script://./bin/test1.sh]
interval = 300
disabled = 1
index = test
sourcetype = doms_test_cpu

As for my outputs.conf (because we are running in a clustered enviroment)

[indexer_discovery:idx_discovery]
pass4SymmKey = xxx
master_uri = xxx

[tcpout:auto_lb_group]
indexerDiscovery = idx_discovery
autoLBFrequency = 30
forceTimebasedAutoLB = true
useACK = true

[indexer_discovery:prd_idx]
pass4SymmKey = xxx
master_uri = xxx

[tcpout:auto_lb_group_prd]
indexerDiscovery = prd_idx
autoLBFrequency = 30
forceTimebasedAutoLB = true
useACK = true

[tcpout]
defaultGroup = auto_lb_group_prd, auto_lb_group

[manjunathmeti]

Command output should be echoed.

Add below line to end of your script and check.

echo "Cmd = [$CMD];  | $AWK '$HEADERIZE $PARSE_0 $PRINTF' header=\"$HEADER\""

[isaacsu]

We tried the default script yesterday, but unfortunately it isn't working.

[peirano]

Hi
I would consider that a possible reason for the different result might be related to quote escaping.
In any case if you are willing just to get the idle time then I would suggest:

mpstat  -u | awk '/\sall\s/ {print "CPU_idle " $13}'

[isaacsu]

Thanks for the suggestion, Peirano. Unfortunately, the target "Suse" server which our forwarder agents are currently installed does not support the "Mpstat" command.

[sumanssah · Accepted solution]

I am not good with the script, however just to apprise to monitor CPU usage on Unix box you can use "Splunk Add-on for Unix and Linux"

https://splunkbase.splunk.com/app/833/

You can also find a script for CPU monitoring file and script in /bin/cpu.sh

[isaacsu]

Thanks for the tips, though this answers half my question as i still need to build a custom script to pull the data due to the target servers limitation.At such, I can't use the /bin/cpu.sh. However, It seems that the default props.conf and transforms.conf from "Splunk Add-on for Unix and Linux" seems to be working it's magic as Splunk is able to interpret and format the _raw data that i have accordingly to what i wanted.

New Script
HEADER='CPU pctIdle'
HEADERIZE="BEGIN {print \"$HEADER\"}"
PRINTF='END {printf "%-3s %9.1f\n", CPU, pctIdle}'
AWK=awk
CMD='vmstat'
PARSE_0='NR==3 {CPU="all"} NR==3 {pctIdle=$15}'

[sumanssah]

I am glad shared details, helped with the completion of your task.