topic Re: Chart disk space over multiple servers in one pie chart in Splunk Search

Chart disk space over multiple servers in one pie chart

jackpal — Wed, 17 Feb 2016 20:47:23 GMT

I need to track disk space over multiple servers in one pie chart. I want to match all volumes with terms in them according to product lines. In my case this is GAS, GEN, STM. The volumes are spread out across multiple machines. I created an event type for each volume match in "df". My search is below.

index=os OR index=main sourcetype=df host=aaaa OR host=bbbb OR host=cccc OR host=dddd eventtype=*Volume NOT eventtype=DBAVolume NOT eventtype=T4STempVolume | strcat host '@' Filesystem Host_FileSystem    | chart  avg(UsePct) by eventtype | rename  avg(UsePct) as %Used

My issue is that "df" is an event type so it takes up a good part of the chart.

Re: Chart disk space over multiple servers in one pie chart

richgalloway — Wed, 17 Feb 2016 21:40:24 GMT

What happens when you add NOT eventtype=df to your base search?

Can we see some sample data and output?

Re: Chart disk space over multiple servers in one pie chart

somesoni2 — Wed, 17 Feb 2016 21:59:39 GMT

Also, what is the definition of the eventtypes you're using? You may be able to write an eval-case to generate a new field denoting volumes accordingly.

Re: Chart disk space over multiple servers in one pie chart

jackpal — Thu, 18 Feb 2016 12:39:36 GMT

using NOT eventtype=df excludes all of the data I am interested in so its not useful.

Re: Chart disk space over multiple servers in one pie chart

jackpal — Thu, 18 Feb 2016 12:44:51 GMT

Event definition for one volume is as follows:

index=os OR index=main sourcetype=df host=aaaa OR host=bbbb OR host=cccc OR host=dddd Filesystem="/dev/mapper/gas"

Re: Chart disk space over multiple servers in one pie chart

somesoni2 — Thu, 18 Feb 2016 15:27:17 GMT

I've seen queries perform better without eventtype being used, so I would suggest (for this query at least) to remove eventtype and use it definition directly. Something like this (change the expansion of eventtype per your environment)

 index=os OR index=main sourcetype=df host=aaaa OR host=bbbb OR host=cccc OR host=dddd 
(Filesystem="/dev/*" ) NOT (Filesystem="/dev/mapper/dba" OR Filesystem="/dev/mapper/t4stemp" ) | strcat host '@' Filesystem Host_FileSystem    | chart  avg(UsePct) by Host_FileSystem  | rename  avg(UsePct) as %Used

Update

To group similar FileSystem into one category (similar to your event type definition), try like this

  index=os OR index=main sourcetype=df host=aaaa OR host=bbbb OR host=cccc OR host=dddd 
    (Filesystem="/dev/*" ) NOT (Filesystem="/dev/mapper/dba" OR Filesystem="/dev/mapper/t4stemp" ) | strcat host '@' Filesystem Host_FileSystem    | chart  avg(UsePct) as avgUsePct by Host_FileSystem  | eval FileSystem=case(like(Host_FileSystem,"%@/dev/mapper/gas%), "GasVolume", like(Host_FileSystem,"%@/dev/mapper/cadbas%), "CADBASVolume") | stats avg(avgUsePct) as "%Used" by FileSystem

Please check the values for Filesystem in case per your situation and add more conditions.

Re: Chart disk space over multiple servers in one pie chart

jackpal — Thu, 18 Feb 2016 16:09:20 GMT

Thanks and understood but using Host_FileSystem now breaks out each volume when the point of the report is roll them all up under their own category. The resulting pie chart now has each individual volume listed (which is quite a bit of them)

Events sample now (too many to list:
1 aaaa/dev/mapper/tcprgas002vg-tcprgas002vol 94.000000
2 aaaa/dev/mapper/tcprgas003vg-tcprgas003vol 100.000000
3 aaaa/dev/mapper/tcprgas005vg-tcprgas005vol 60.000000
4 aaaa/dev/mapper/tcprgas006vg-tcprgas006vol 1.000000

Events sample before when broken out by event type:
1 CADBASVolume 1.000000
2 GasVolume 88.461538
3 GeneratorVolume 44.000000
4 SPGVolume 85.333333
5 SteamVolume 49.666667
6 df 81.698113

Re: Chart disk space over multiple servers in one pie chart

somesoni2 — Thu, 18 Feb 2016 17:13:52 GMT

Well, that's the reason I asked for event type definition (Settings->Event types -> Your event type [Search string]), which can be used in a case statatement to summarize this Host_FileSystem into logical names similar to your event types.

Re: Chart disk space over multiple servers in one pie chart

jackpal — Thu, 18 Feb 2016 17:59:55 GMT

Sorry I thought you saw that in one of the above posts.

index=os OR index=main sourcetype=df host=aaaa OR host=bbbb OR host=cccc OR host=dddd Filesystem="/dev/mapper/gas"

Thanks for the help.

Re: Chart disk space over multiple servers in one pie chart

jackpal — Tue, 22 Mar 2016 19:04:09 GMT

With some extra help I was able to do the following. Thanks to all for their help on this thread.

index=os sourcetype=df host=aaaa OR host=bbbb 
filesystem="/dev/mapper/tcpr*" |
eval CPD_Disk=case(
filesystem LIKE "/dev/mapper/tcprgas%", "Gas Volume",
filesystem LIKE "/dev/mapper/tcprcadbas%", "CADBAS Volume",
filesystem LIKE "/dev/mapper/tcprspg%", "SPG Volume",
filesystem LIKE "/dev/mapper/tcprgen%", "Generator Volume",
filesystem LIKE "/dev/mapper/tcprstm%", "Steam Volume"
)
| chart eval(sum(UsedMBytes)/1024/1024) as TerraBytes by CPD_Disk