All Apps and Add-ons

multikv command isn't assigning columns correctly - sourcetype=iostat

mikelanghorst
Motivator

I was trying to use splunk to filter iostat information on a Linux host, but found that it isn't assigning columns properly when pointing to a RedHat system, but works as expected on Solaris. When passing the Linux output to multikv each disk device is it's own field name.

Data samples
Solaris (works as expected):

Device          rReq_PS      wReq_PS        rKB_PS        wKB_PS  avgWaitMillis   avgSvcMillis   bandwUtilPct
sd1                 0.0          0.0           0.0           0.0            0.0            0.0              0
sd2                 0.0          0.0           0.0           0.0            0.0            0.0              0
sd46                0.0          0.0           0.0           0.0            0.0            0.0              0
ssd24               0.0          0.0           0.0           0.0            0.0            0.0              0

Linux:

Device          rReq_PS      wReq_PS        rKB_PS        wKB_PS  avgWaitMillis   avgSvcMillis   bandwUtilPct
sda                0.00         1.98          0.00          0.00           0.50           0.50           0.10
sda1               0.00         0.00          0.00          0.00           0.00           0.00           0.00
sda2               0.00         0.00          0.00          0.00           0.00           0.00           0.00
sda3               0.00         0.00          0.00          0.00           0.00           0.00           0.00

The data looks similar, though the Linux host I'm attempting to look at has many more devices than the Solaris samples I've tried. Anyone know why this could be occurring?

asimagu
Builder

hey, hey! I think I came up with something that works

First, you need to tweak the iostat.sh . Make sure you edit the last 2 lines of the script

$CMD | tee $TEE_DEST | awk "$HEADERIZE $FILTER $FORMAT $PRINTF" header="$HEADER" | while read 'line'; do if [ echo "$line" |wc -w -gt 1 ]; then echo "$lastline" "$line"; lastline=""; else lastline=$line ; fi; done

echo "Cmd = [$CMD]; | awk '$HEADERIZE $FILTER $FORMAT $PRINTF' header=\"$HEADER\"" >> $TEE_DEST

You need to be careful with the single and double quotes coz otherwise it won't work.
Try the script manually by running it on the shell

Then on Splunk you should be able to multikv all the fields properly now.

PS: make sure you are splunking on Verbose Mode, in order to watch the extracted field names on the left bar

happy splunkin'!

mikelanghorst
Motivator

When I run "$splunk_home/etc/apps/unix/bin/iostat.sh > /tmp/iostat.out", and then vi the resulting file, the output for long rows isn't wrapped but is an additional line. Other hosts which get their results "fielded" properly have single lines.

Still to find out why this is occurring on only some hosts.

Narrowing down a bit further, iostat.sh on linux executes "iostat -xk 1 2". If I output that to a file, it has several of the rows in multiple lines as well.

mikelanghorst
Motivator

Never hurts to post it up here for others to see as well. Even when you eventually may find it's user error.

0 Karma

araitz
Splunk Employee
Splunk Employee

Mr. Langhorst usually troubleshoots his own issues! Bonus Karma.

0 Karma

mikelanghorst
Motivator

Looking around further, it doesn't look to be a Solaris vs. Linux issue, but rather the number of results returned by the iostat command.

The host that's breaking this data, iostat returns 2000+ lines: "iostat 1 1 | wc -l"

0 Karma

lguinn2
Legend

I'd also look to see if the whitespace is different between the two. I've seen this before on other things - where one input inserts spaces and the other uses tabs...

0 Karma

jrodman
Splunk Employee
Splunk Employee

I'd want to start by narrowing down the dataset to see if the problem exists with a single Linux event or the set in aggregate somehow.

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.