How to create a timechart dashboard to monitor variations in file system percentage?

[thaara]

I am trying to create a dashboard to monitor the file system for two different hosts. My ultimate aim is to create a time chart dashboard which shows any variations in file system percentage. My requirements are as follows:

• Hosts as legends
• Timestamp in x-axis
• File system usage in y-axis

Note: I have to search for any required file system in the query.

Sample query I created:

index= index1 sourcetype=stype Mon=path1 OR path2 OR path3 |eval UPT=replace(UPT,"%","") | chart values(UPT) as UPT by Mon, host 

Please do let me know what can be done to set up these requirements in Splunk.

[richgalloway]

What is "Mon" and how is it different from host? What are you trying to monitor from these file systems? Disk usage or something else? Is the data in Splunk?

[thaara]

Hi @richgalloway

'Mon, represents the file system(eg. /apps, /logs, /opt). We want to monitor disk usage of selective file systems from the data we have.
Data is already on boarded under Splunk in below format.

Filesystem,Size,Used,Avail,Use%,Mounted,on
/dev/mapper/rootvg-lv_opt,30G,20G,8.4G,71%,/opt
/dev/mapper/rootvg-lv_tmp,9.8G,759M,8.6G,8%,/tmp
/dev/mapper/rootvg-lv_var,9.8G,6.7G,2.6G,73%,/var
/dev/mapper/appsvg-lv_apps,89G,60G,25G,72%,/apps

By using the query mentioned in description, we are able to create chart dashboard. But we need timechart dashboard which shows variation in each file system host wise.

Let me know if you need any further details.

[ololdach]

Hi thaara,
there are two things that you want to solve:
- the aggregation of the values by time
- the aggregation of values by host
For the host aggregation, you have to define how to interpret the different fill levels per host into one "health" score. For example, if your / is at 45%, but your /tmp is at 99%... does that translate into about 75% (the average) or 99% (the value of the worst fs). You get the point... if you only want one value per host, how do you calculate it in order to give you the "actionable hosts" that you need to take care of?
The time aggregation is straight forward:
index=index1 sourcetype=stype Mon IN ("path1","path2","path3) | bin _time span=1m (put your time into discrete slots) | eval health=(whatever you do to calculate the heatlh score from your Mon/UPT pairs) | stats avg(health) as aheatlh by _time, host
This should help you get this sorted.
Oliver

[thaara]

Hi @ololdach ,

thanks for the suggestion.

But we want the same in multiseries, Like host wise we need this..
Filesystem(in chart ) vs time(x-axis) vs values(y-axis)

host-1 chart:
Filesystem(in chart ) vs time(x-axis) vs values(y-axis)

host -2 chart:
Filesystem(in chart ) vs time(x-axis) vs values(y-axis)

both charts should be in single dashboard..

[thaara]

@ololdach Multi-series timechart is my requirement...

[ololdach]

Hi thaara,
my suggestion still works, just omit the part with the health score, leave the individual paths in, do the stats ... by _time, host, Mon and in your visualisation select "trellis" with split by host