topic Re: Disk Space Pie Charts from "df" Script in UNIX App in Splunk Search

Disk Space Pie Charts from "df" Script in UNIX App

aferone — Tue, 15 Jan 2013 18:31:52 GMT

I'd like to set up pie charts for disk space from data coming from the "df" scripts from the UNIX app. In looking through the charting docs, I can see how to chart "count" data, but I am unclear how to chart a single value, coming from the "df" script. I tried the "transpose" command, but only "UsedG" is coming up on the chart.

The search below was developed with help from the Splunk Answers KB. Notice that we convert the "Used" field to gigabytes. It goes to a field called "UsedG", but this seems to screw up the table. It adds "UsedG" to the table, even though it is not defined in the search.

I basically want to use "Used" and "Available", which should give a complete pie chart. If there was a way to chart "Size" and "Used", that would probably be more accurate.

Thanks for the help!

host="myhost" sourcetype="df" 
| multikv fields Filesystem Type Size Used Avail UsePct MountedOn 
| search /home | table MountedOn Size Used Avail 
| eval UsedG = case(match(Used,"[M]"),round(tonumber(rtrim(Used,"M"))/1024,3),
       match(Used,"[T]"),round(tonumber(rtrim(Used,"T"))*1024,3),
       match(Used,"[G]"),round(tonumber(rtrim(Used,"G")),3))

Re: Disk Space Pie Charts from "df" Script in UNIX App

christopher_hod — Tue, 15 Jan 2013 20:16:07 GMT

For starters, use |convert to normalize those values instead of that eval.

I would also drop that table command.

And you probably only want the most recent value, but you want all filesystems.

Finally, you need to actually chart the data.

So what we're left with is something like this:
host="myhost" sourcetype="df"
| head 1
| multikv fields Filesystem Type Size Used Avail UsePct MountedOn
| convert memk(Used)
| chart avg(Used) AS Used by Filesystem

Re: Disk Space Pie Charts from "df" Script in UNIX App

lguinn2 — Tue, 15 Jan 2013 20:24:53 GMT

(Updated with convert command instead of eval, and explanation - also updated to address "every 5 minute" problem)
My first question is - so why do you calculate UsedG if you never want to use it?

This will get your pie chart

host="myhost" sourcetype="df" earliest=-10m
| multikv fields Used Avail MountedOn
| search /home 
| dedup MountedOn
| eval s = "Used,Available"
| makemv delim="," allowempty=t s
| mvexpand s
| eval Size = if(s=="Used",Used,Avail)
| convert memk(Size) as Size 
| chart sum(Size) as "Size in Gb" by s

Try this for a column chart:

host="myhost" sourcetype="df" 
| multikv fields Filesystem Type Size Used Avail UsePct MountedOn 
| dedup MountedOn
| eval s = "Used,TotalSize"
| makemv delim="," allowempty=t s
| mvexpand s
| eval Size = if(s=="Used",Used,Size)
| convert memk(Size) as Size 
| chart sum(Size) as "Size in Gb" by MountedOn, s

Explanation by line -

host="myhost" sourcetype="df"

| multikv fields Filesystem Type Size Used Avail UsePct MountedOn

The initial search and field extraction

| eval s = "Used,TotalSize"

| makemv delim="," allowempty=t s

| mvexpand s

Create a new variable s that contains 2 values. Tell Splunk to consider this a multi-valued field. Expand this event into two events, one for each value of s. Except for that, the events are the same. This turns s back into a single-valued field, but creates multiple events.

| eval Size = if(s=="Used",Used,Size)

Set the variable Size to the amount of disk used, when the field s refers to "used". Otherwise, the field is the total size, so use the original Size field

| convert memk(Size) as Size

Convert the Size into a true numeric field, representing KB

| chart sum(Size) as "Size in Gb" by MountedOn, s

Chart the size field, breaking it down by mount point, and within that by Used and TotalSize. If you chart this using a column chart, you can see the two bars side-by-side.

Re: Disk Space Pie Charts from "df" Script in UNIX App

aferone — Tue, 15 Jan 2013 20:37:45 GMT

I actually used "UsedG" for a different report.

I need to lookup some of the terms you used. I've never seen them before. I am still a noob in many ways!

This works great! Thanks!

Re: Disk Space Pie Charts from "df" Script in UNIX App

aferone — Tue, 15 Jan 2013 20:38:30 GMT

Thanks for your answer! I learned some new terms from your input. Thanks again!

Re: Disk Space Pie Charts from "df" Script in UNIX App

aferone — Tue, 15 Jan 2013 20:41:50 GMT

Can you just quickly breakdown how this works? I appreciate it!

Re: Disk Space Pie Charts from "df" Script in UNIX App

lguinn2 — Tue, 15 Jan 2013 23:19:07 GMT

Nice use of the memk function!

Re: Disk Space Pie Charts from "df" Script in UNIX App

aferone — Wed, 16 Jan 2013 16:21:43 GMT

Thank you!!

Re: Disk Space Pie Charts from "df" Script in UNIX App

aferone — Wed, 16 Jan 2013 16:39:57 GMT

So, now that I matching up the numbers from the chart and the actual log, they aren't matching up. I'm getting completely different results. The chart works, but it's the wrong values entirely. The search below christipherhodson brings back the right values, but just the Used values, and not total disk space.

Re: Disk Space Pie Charts from "df" Script in UNIX App

lguinn2 — Wed, 16 Jan 2013 20:31:41 GMT

Weird, sounds like maybe something is up with the multikv

You might try just running this to see if the extracted values are correct:

host="myhost" sourcetype="df" | multikv fields Filesystem Type Size Used Avail UsePct MountedOn | table fields Filesystem Type Size Used Avail UsePct MountedOn

Re: Disk Space Pie Charts from "df" Script in UNIX App

aferone — Wed, 16 Jan 2013 20:35:03 GMT

Thanks for replying again. Yes, all the values are correct.

Re: Disk Space Pie Charts from "df" Script in UNIX App

aferone — Wed, 16 Jan 2013 20:37:17 GMT

I am going to post another "answer" below so that I can provide screenshots.

Re: Disk Space Pie Charts from "df" Script in UNIX App

aferone — Wed, 16 Jan 2013 20:39:48 GMT

This is what "df" is returning:

And this is what the query returns:

Re: Disk Space Pie Charts from "df" Script in UNIX App

aferone — Wed, 16 Jan 2013 20:41:08 GMT

It actually ended up above, since it is my own question.

Re: Disk Space Pie Charts from "df" Script in UNIX App

aferone — Wed, 16 Jan 2013 20:54:52 GMT

I think I see what is happening. I had the time frame set to 1 hour, and the "df" command runs every 5 minutes. the query is adding together all of the "df" findings. How would I pull only the latest record, without having to rely on a timeframe?

Re: Disk Space Pie Charts from "df" Script in UNIX App

christopher_hod — Thu, 17 Jan 2013 12:25:26 GMT

That's why I used the "|head 1" in my answer below. You could also use the latest() function.

FYI, if you want to use a pie chart, you're only going to want one value per filesystem.

You could also do one pie chart per filesystem, but that's more complex

Re: Disk Space Pie Charts from "df" Script in UNIX App

aferone — Thu, 17 Jan 2013 15:10:03 GMT

Cool. Thanks, Chris!

Re: Disk Space Pie Charts from "df" Script in UNIX App

abhay24 — Wed, 20 Jul 2016 15:54:14 GMT

Do we need to write a stanza seperately for inputs.conf in the host server ?
if so can you send me the stanza for getting df -h command data and ls -l command data.