I was trying to use splunk to filter iostat information on a Linux host, but found that it isn't assigning columns properly when pointing to a RedHat system, but works as expected on Solaris. When passing the Linux output to multikv each disk device is it's own field name.
Solaris (works as expected):
Device rReq_PS wReq_PS rKB_PS wKB_PS avgWaitMillis avgSvcMillis bandwUtilPct sd1 0.0 0.0 0.0 0.0 0.0 0.0 0 sd2 0.0 0.0 0.0 0.0 0.0 0.0 0 sd46 0.0 0.0 0.0 0.0 0.0 0.0 0 ssd24 0.0 0.0 0.0 0.0 0.0 0.0 0
Device rReq_PS wReq_PS rKB_PS wKB_PS avgWaitMillis avgSvcMillis bandwUtilPct sda 0.00 1.98 0.00 0.00 0.50 0.50 0.10 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00
The data looks similar, though the Linux host I'm attempting to look at has many more devices than the Solaris samples I've tried. Anyone know why this could be occurring?
hey, hey! I think I came up with something that works
First, you need to tweak the iostat.sh . Make sure you edit the last 2 lines of the script
$CMD | tee $TEE_DEST | awk "$HEADERIZE $FILTER $FORMAT $PRINTF" header="$HEADER" | while read 'line'; do if [
echo "$line" |wc -w -gt 1 ]; then echo "$lastline" "$line"; lastline=""; else lastline=$line ; fi; done
echo "Cmd = [$CMD]; | awk '$HEADERIZE $FILTER $FORMAT $PRINTF' header=\"$HEADER\"" >> $TEE_DEST
You need to be careful with the single and double quotes coz otherwise it won't work.
Try the script manually by running it on the shell
Then on Splunk you should be able to multikv all the fields properly now.
PS: make sure you are splunking on Verbose Mode, in order to watch the extracted field names on the left bar
When I run "$splunk_home/etc/apps/unix/bin/iostat.sh > /tmp/iostat.out", and then vi the resulting file, the output for long rows isn't wrapped but is an additional line. Other hosts which get their results "fielded" properly have single lines.
Still to find out why this is occurring on only some hosts.
Narrowing down a bit further, iostat.sh on linux executes "iostat -xk 1 2". If I output that to a file, it has several of the rows in multiple lines as well.
Looking around further, it doesn't look to be a Solaris vs. Linux issue, but rather the number of results returned by the iostat command.
The host that's breaking this data, iostat returns 2000+ lines: "iostat 1 1 | wc -l"