Getting Data In

Linux Script output is different from _raw

Explorer

Hi all,

I've written script in "/opt/splunk/etc/deployment-apps/testing/bin" named test1.sh which gives me an expected result what i intend the script to be used for.

Script Content.

#!/bin/sh

HEADER='CPU    pctIdle'
HEADERIZE="BEGIN {print \"$HEADER\"}"
PRINTF='END {printf "%-3s  %9.1f\n", CPU, pctIdle}'
AWK=awk
CMD='top -n 1'
PARSE_0='NR==3 {CPU="all"} NR==3 {pctIdle=$8}'

$CMD | $AWK "$HEADERIZE $PARSE_0 $PRINTF"  header="$HEADER"

Script Results

CPU    pctIdle
all       94.9

However, upon running "/opt/splunk/bin/splunk reload deploy-server". The _ raw values of which i receive from my events are all as such.

CPU    pctIdle
           0.0

Can i get some insights on why is this so?

0 Karma
1 Solution

Communicator

I am not good with the script, however just to apprise to monitor CPU usage on Unix box you can use "Splunk Add-on for Unix and Linux"

https://splunkbase.splunk.com/app/833/

You can also find a script for CPU monitoring file and script in /bin/cpu.sh

View solution in original post

Esteemed Legend

Generally what happens is that when you run it manually, you are receiving the benefit of the PATH environment variable associated with your user. However, when Splunk runs it as user = splunk, this user has a different value for PATH or maybe no value at all, which means that in your scripts, you should *ALWAYS* fully-qualify the path to every command. So your script would look something like this:

#!/bin/sh
HEADER='CPU    pctIdle'
HEADERIZE="BEGIN {print \"$HEADER\"}"
PRINTF='END {printf "%-3s  %9.1f\n", CPU, pctIdle}'
AWK="/usr/bin/awk"
CMD='/usr/bin/top -n 1'
PARSE_0='NR==3 {CPU="all"} NR==3 {pctIdle=$8}'
$CMD | $AWK "$HEADERIZE $PARSE_0 $PRINTF"  header="$HEADER"

Your coding is very awkwardly convoluted. It should be half as long with no variables at all.

0 Karma

Explorer

Hi Woodcock, based on our observations. We've tried to define both the full path and also the path you've seen in the script above. Unfortunately, both produced the same results. Also, apologies for the convoluted script, i'm still new to this ^_^l

0 Karma

Ultra Champion

If you are just interested in CPU data I echo the sentiments below - use the Splunk provided TA as all of this work is done for you.

If, however this is an exercise which you plan to expand further, you can make you Splunk ingestion journey much easier using K=V values as the response from your script. Also make you your life easier by including a timestamp.

CSV output formats make a huge amount of sense if the output from your script is large and repetitive. In those cases you only write the field names (header) out once, and then list all the values, but for a simple output from a script, you will find it much easier if you format your output like so:

06/03/2020 09:36:24 cpu=all pctIdle=94.6
Splunk will automatically break events formatted like this, and the fields will be auto extracted for you

0 Karma

Influencer

And where is app/add-on testing is deployed?

0 Karma

Explorer

This is currently being deployed via deployment-apps in my Cluster Master

0 Karma

Influencer

Is this script part of scripted input? And how the output of script is forwarded to indexer server?

0 Karma

Explorer

The script i wrote resides in the bin folder of my testing app "/opt/splunk/etc/deployment-apps/testing/bin".

It is being invoked by "/opt/splunk/etc/deployment-apps/testing/local" through the following inputs.conf stanza.

[script://./bin/test1.sh]
interval = 300
disabled = 1
index = test
sourcetype = doms_test_cpu

As for my outputs.conf (because we are running in a clustered enviroment)

[indexer_discovery:idx_discovery]
pass4SymmKey = xxx
master_uri = xxx

[tcpout:auto_lb_group]
indexerDiscovery = idx_discovery
autoLBFrequency = 30
forceTimebasedAutoLB = true
useACK = true

[indexer_discovery:prd_idx]
pass4SymmKey = xxx
master_uri = xxx

[tcpout:auto_lb_group_prd]
indexerDiscovery = prd_idx
autoLBFrequency = 30
forceTimebasedAutoLB = true
useACK = true

[tcpout]
defaultGroup = auto_lb_group_prd, auto_lb_group
0 Karma

Influencer

Command output should be echoed.

Add below line to end of your script and check.

echo "Cmd = [$CMD];  | $AWK '$HEADERIZE $PARSE_0 $PRINTF' header=\"$HEADER\""
0 Karma

Explorer

We tried the default script yesterday, but unfortunately it isn't working.

0 Karma

Engager

Hi
I would consider that a possible reason for the different result might be related to quote escaping.
In any case if you are willing just to get the idle time then I would suggest:

mpstat  -u | awk '/\sall\s/ {print "CPU_idle " $13}'

Explorer

Thanks for the suggestion, Peirano. Unfortunately, the target "Suse" server which our forwarder agents are currently installed does not support the "Mpstat" command.

0 Karma

Communicator

I am not good with the script, however just to apprise to monitor CPU usage on Unix box you can use "Splunk Add-on for Unix and Linux"

https://splunkbase.splunk.com/app/833/

You can also find a script for CPU monitoring file and script in /bin/cpu.sh

View solution in original post

Explorer

Thanks for the tips, though this answers half my question as i still need to build a custom script to pull the data due to the target servers limitation.At such, I can't use the /bin/cpu.sh. However, It seems that the default props.conf and transforms.conf from "Splunk Add-on for Unix and Linux" seems to be working it's magic as Splunk is able to interpret and format the _raw data that i have accordingly to what i wanted.

New Script
HEADER='CPU pctIdle'
HEADERIZE="BEGIN {print \"$HEADER\"}"
PRINTF='END {printf "%-3s %9.1f\n", CPU, pctIdle}'
AWK=awk
CMD='vmstat'
PARSE_0='NR==3 {CPU="all"} NR==3 {pctIdle=$15}'

0 Karma

Communicator

I am glad shared details, helped with the completion of your task.

0 Karma