multikv command isn't assigning columns correctly - sourcetype=iostat

[mikelanghorst]

I was trying to use splunk to filter iostat information on a Linux host, but found that it isn't assigning columns properly when pointing to a RedHat system, but works as expected on Solaris. When passing the Linux output to multikv each disk device is it's own field name.

Data samples
Solaris (works as expected):

Device          rReq_PS      wReq_PS        rKB_PS        wKB_PS  avgWaitMillis   avgSvcMillis   bandwUtilPct
sd1                 0.0          0.0           0.0           0.0            0.0            0.0              0
sd2                 0.0          0.0           0.0           0.0            0.0            0.0              0
sd46                0.0          0.0           0.0           0.0            0.0            0.0              0
ssd24               0.0          0.0           0.0           0.0            0.0            0.0              0

Linux:

Device          rReq_PS      wReq_PS        rKB_PS        wKB_PS  avgWaitMillis   avgSvcMillis   bandwUtilPct
sda                0.00         1.98          0.00          0.00           0.50           0.50           0.10
sda1               0.00         0.00          0.00          0.00           0.00           0.00           0.00
sda2               0.00         0.00          0.00          0.00           0.00           0.00           0.00
sda3               0.00         0.00          0.00          0.00           0.00           0.00           0.00

The data looks similar, though the Linux host I'm attempting to look at has many more devices than the Solaris samples I've tried. Anyone know why this could be occurring?

[asimagu]

hey, hey! I think I came up with something that works

First, you need to tweak the iostat.sh . Make sure you edit the last 2 lines of the script

$CMD | tee $TEE_DEST | awk "$HEADERIZE $FILTER $FORMAT $PRINTF" header="$HEADER" | while read 'line'; do if [ echo "$line" |wc -w -gt 1 ]; then echo "$lastline" "$line"; lastline=""; else lastline=$line ; fi; done

echo "Cmd = [$CMD]; | awk '$HEADERIZE $FILTER $FORMAT $PRINTF' header=\"$HEADER\"" >> $TEE_DEST

You need to be careful with the single and double quotes coz otherwise it won't work.
Try the script manually by running it on the shell

Then on Splunk you should be able to multikv all the fields properly now.

PS: make sure you are splunking on Verbose Mode, in order to watch the extracted field names on the left bar

happy splunkin'!

[mikelanghorst]

When I run "$splunk_home/etc/apps/unix/bin/iostat.sh > /tmp/iostat.out", and then vi the resulting file, the output for long rows isn't wrapped but is an additional line. Other hosts which get their results "fielded" properly have single lines.

Still to find out why this is occurring on only some hosts.

Narrowing down a bit further, iostat.sh on linux executes "iostat -xk 1 2". If I output that to a file, it has several of the rows in multiple lines as well.

[mikelanghorst]

Never hurts to post it up here for others to see as well. Even when you eventually may find it's user error.

[araitz]

Mr. Langhorst usually troubleshoots his own issues! Bonus Karma.

[mikelanghorst]

Looking around further, it doesn't look to be a Solaris vs. Linux issue, but rather the number of results returned by the iostat command.

The host that's breaking this data, iostat returns 2000+ lines: "iostat 1 1 | wc -l"

[lguinn2]

I'd also look to see if the whitespace is different between the two. I've seen this before on other things - where one input inserts spaces and the other uses tabs...

[jrodman]

I'd want to start by narrowing down the dataset to see if the problem exists with a single Linux event or the set in aggregate somehow.