Drive usage over time

[fhemmeld]

I created a simple search:

index=index1 sourcetype="Perfmon:Free Disk Space" instance="D:\drive\drive_01"
| timechart span=1d max(Value)

It shows me the amount of available free space over a certain period. Very useful to determine how fast a drive is filling up. The disadvantage is you have to find the time period in which the drive went from 100% free space to 10% free space (the system leaves some space on the drive before switching to the next drive, so it will never get to 0%).

I would like to create a dashboard that will show a graph of the drive and a pull down menu, based on a lookup file (or similar) that allows my Splunk users to look at a drive and see the rate of disk usage over time.

I am not an advanced Splunk user, hence posting this question. I can see the process:

- Select drive from pulldown list
- For selected drive, find point in time where % free space is 98%
- For selected drive, find point in time where % free space is 10% or current date/time (if not yet at 10%)
- Display line chart graph for the period found

Not sure something like that would be possible, but a question not asked is an answer missed 🙂

Thank you!

[bowesmana · Accepted solution]

@fhemmeld 

I put together an example dashboard that shows the sort of behaviour you are trying to achieve. I have done it with randomised data and have created fictitious starting and ending disk free capacity, which will give different numbers each time you refresh the dashboard.

The idea is is that you can see how the use of dropdown inputs, searches and in particular, how tokens are set and used, to hopefully get your search working.

I have also used a 'base' search, which is a common search used by other search panels.

One question you will have to resolve is how you want to address the time range of how to 'discover' when a drive was in use? The input box for drive selection would typically search over a time range you set, or pull drives from a lookup, but the lifetime of that drive may not be known, so it it was present last month and you select this month in the search range, then it will not know about it.

Anyway, I hope this dashboard example will give you an idea of what you can do - the answer with Splunk is that it always can do what you want, you just need to work out how 🙂

Create a new dashboard and copy this example below into the dashboard source.

Hope this is useful

<form>
  <label>Disks</label>
  <search id="base">
    <!--index=index1 sourcetype="Perfmon:Free Disk Space" instance=$disk$-->
    <query>
| makeresults
| eval n=mvrange(1,169)
| eval instance="D:\drive\drive_01"
| eval Value=random() % 7 + 94
| eval drop=(random() % 100 / 100)
| eval _time=_time-(168*3600)
| mvexpand n
| eval _time=_time+(n-1)*3600
| eval Value=Value-(n-1)*drop, Value=max(Value,10)
| fields - drop
| search instance=$disk|s$
    </query>
    <earliest>$time_range.earliest$</earliest>
    <latest>$time_range.latest$</latest>
  </search>
  <fieldset submitButton="false">
    <input type="time" token="time_range" searchWhenChanged="true">
      <label>Max time range</label>
      <default>
        <earliest>-7d@h</earliest>
        <latest>now</latest>
      </default>
    </input>
    <input type="dropdown" token="disk" searchWhenChanged="true">
      <label>Disk</label>
      <fieldForLabel>instance</fieldForLabel>
      <fieldForValue>instance</fieldForValue>
      <search>
        <query>index=index1 sourcetype="Perfmon:Free Disk Space" instance=* earliest=$time_range.earliest$ latest=$time_range.latest$
| stats count by instance</query>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </search>
      <choice value="D:\drive\drive_01">D:\drive\drive_01</choice>
    </input>
  </fieldset>
  <row>
    <panel>
      <chart>
        <title>Disk free space over selected time range</title>
        <search base="base">
          <query>
| timechart span=1d max(Value)</query>
        </search>
        <option name="charting.chart">line</option>
        <option name="charting.drilldown">none</option>
        <option name="refresh.display">progressbar</option>
      </chart>
    </panel>
    <panel>
      <table>
        <title>Date when threshold free space went under 98%</title>
        <search base="base">
          <query>
| where Value&lt;=98
| table _time Value
| sort - Value
| head 1
| eval thresholdReached=if(round(Value)=98,null(),"Threshold started under 98%")
| eval date=strftime(_time, "%F")
| eval t=_time
          </query>
          <done>
            <condition match="$job.resultCount$ == 0">
              <set token="start_time">$time_range.earliest$</set>
              <set token="start_date">N/A</set>
              <set token="start_free">N/A</set>
              <set token="start_threshold">N/A</set>
            </condition>
            <condition match="$job.resultCount$ &gt; 0">
              <set token="start_time">$result.t$</set>
              <set token="start_date">$result.date$</set>
              <set token="start_free">$result.Value$</set>
              <eval token="start_threshold">if(isnull($result.thresholdReached$),null(),$result.thresholdReached$)</eval>
            </condition>
          </done>
        </search>
        <fields>_time, Value</fields>
        <option name="drilldown">none</option>
        <option name="refresh.display">progressbar</option>
      </table>
      <single>
        <title>Date when threshold free space went under 98%</title>
        <search>
          <query>| makeresults 
| eval date="$start_date$"
| table date</query>
          <earliest>$earliest$</earliest>
          <latest>$latest$</latest>
        </search>
        <option name="drilldown">none</option>
        <option name="rangeColors">["0x53a051","0x0877a6","0xf8be34","0xf1813f","0xdc4e41"]</option>
        <option name="refresh.display">progressbar</option>
        <option name="showSparkline">0</option>
        <option name="showTrendIndicator">0</option>
      </single>
      <single>
        <title>Threshold at that date</title>
        <search>
          <query>
| makeresults 
| eval threshold="$start_free$"
| table threshold
          </query>
        </search>
        <option name="drilldown">none</option>
        <option name="refresh.display">progressbar</option>
      </single>
      <html depends="$start_threshold$">
        <h1 style="color:red">$start_threshold$</h1>
      </html>
    </panel>
    <panel>
      <title>Date when threshold free space reached 10%</title>
      <table>
        <search base="base">
          <query>
| table _time Value
| sort Value
| head 1
| eval thresholdReached=if(Value&lt;=10,null(),"Threshold not yet at 10%")
| eval date=strftime(_time, "%F")
| eval t=_time
          </query>
          <done>
            <condition match="$job.resultCount$ == 0">
              <set token="end_time">$time_range.latest$</set>
              <set token="end_free">N/A</set>
              <set token="end_threshold">N/A</set>
            </condition>
            <condition match="$job.resultCount$ &gt; 0">
              <set token="end_time">$result.t$</set>
              <set token="end_date">$result.date$</set>
              <set token="end_free">$result.Value$</set>
              <eval token="end_threshold">if(isnull($result.thresholdReached$),null(),$result.thresholdReached$)</eval>
            </condition>
          </done>
        </search>
        <fields>_time, Value</fields>
        <option name="drilldown">none</option>
        <option name="refresh.display">progressbar</option>
      </table>
      <single>
        <title>Date when threshold free space reached 10%</title>
        <search>
          <query>
| makeresults 
| eval date="$end_date$"
| table date
          </query>
        </search>
        <option name="drilldown">none</option>
        <option name="refresh.display">progressbar</option>
      </single>
      <single>
        <title>Threshold at that date</title>
        <search>
          <query>
| makeresults 
| eval threshold="$end_free$"
| table threshold
          </query>
        </search>
        <option name="drilldown">none</option>
        <option name="refresh.display">progressbar</option>
      </single>
      <html depends="$end_threshold$">
        <h1 style="color:red">$end_threshold$</h1>
      </html>
    </panel>
  </row>
  <row>
    <panel>
      <chart>
        <title>Disk free space between max and min</title>
        <search base="base">
          <query>
| where _time&gt;=$start_time$ AND _time&lt;=$end_time$
| timechart fixedrange=f span=1d max(Value)</query>
        </search>
        <option name="charting.chart">line</option>
        <option name="charting.drilldown">none</option>
        <option name="refresh.display">progressbar</option>
      </chart>
    </panel>
  </row>
  <row>
    <panel>
      <title>List of events</title>
      <table>
        <search base="base">
          <query>
| table _time Value
          </query>
        </search>
        <option name="drilldown">none</option>
        <option name="refresh.display">progressbar</option>
      </table>
    </panel>
  </row>
</form>

[fhemmeld]

Thank you @bowesmana, will have a play with it. Much appreciated.

Will let you know how it works out.