topic Re: Multiple statistics for multiple linux servers, How do I write the query? in Splunk Search

Multiple statistics for multiple linux servers, How do I write the query?

mike000 — Wed, 30 Sep 2020 04:49:28 GMT

I tried:
index=_nix_xxxx sourcetype=df host=abdhw003 MountedOn="/doc" |eval source="/doc*" and that seems to show the data of the /doc folder

Now I have multiple servers and I need stats on all the servers seperately whichever server has >5% used. Please see example below. But when i run the below command then All the servers are adding up ad its showing me a 1 liner with all server info merged. I think I messed up in the stats algorithm. Please help.

index=_nix_xxxx sourcetype=df host=abdhw003 OR host=n OR host=n OR host=n or host=n MountedOn="/doc"| eval TotalGBytes= TotalMBytes/1024 | eval UsedGBytes=UsedMbytes/1024 |eval used_pct=100(UsedGBytes/TotalGBytes) | stats max(TotalGBytes) as "MaxSize(GB) max(UsedGBytes) as "UsedSize(GB) as "percentUsed" by MountedOn | search PercentUsed>05| Sort PercentUsed

Now the stats that I am getting is getting totalled(All 5 servers adding each other and showing me a max value) I think as the stats query has max value, How do I show stats of each server at a time? Any Ideas?

Thanks for the help. I appreciate it

Re: Multiple statistics for multiple linux servers, How do I write the query?

richgalloway — Fri, 03 Apr 2020 13:46:17 GMT

Try this:

index=_nix_xxxx sourcetype=df host=abdhw003 OR host=n OR host=n OR host=n or host=n MountedOn="/doc"
| eval TotalGBytes= TotalMBytes/1024 | eval UsedGBytes=UsedMbytes/1024 |eval used_pct=100(UsedGBytes/TotalGBytes) 
| stats max(TotalGBytes) as "MaxSize(GB) max(UsedGBytes) as "UsedSize(GB) as "percentUsed" by host, MountedOn 
| search PercentUsed>05| Sort PercentUsed

Re: Multiple statistics for multiple linux servers, How do I write the query?

mike000 — Fri, 03 Apr 2020 14:01:44 GMT

Thank you soo much, that worked like a charm. Much appreciated Rich, you rock!!!!

Re: Multiple statistics for multiple linux servers, How do I write the query?

woodcock — Fri, 03 Apr 2020 17:00:52 GMT

Try this:

index="_nix_xxxx" AND sourcetype="df" AND (host="abdhw003" OR host="x" OR host="y" OR host="z" or host="etc") AND MountedOn="/doc"
| stats max(TotalMBytes) AS max_TotalMBytes max(UsedMBytes) AS max_UsedMBytes avg(PercentUsedSpace) AS PercentUsed BY MountedOn 
| search PercentUsed>05 
| eval TotalGBytes = max_TotalMBytes / 1024
| eval UsedGBytes = max_UsedMBytes / 1024
| eval used_pct = 100 * (max_UsedMBytes / max_TotalMBytes)
| fields - TotalGBytes UsedGBytes
| sort 0 PercentUsed

As a bonus, try this search, use the Linechart visualization, turn on Trellis feature and select host for split by:

index="os" AND sourcetype="df" 
| eval FS_{Filesystem}=PercentUsedSpace
| timechart limit=0 [search index="os" AND sourcetype="df" earliest=-4h latest=now
| stats count BY Filesystem
| rename Filesystem AS max
| eval max = "FS_" . max
| eval zAS = max
| table max zAS
| format "" "" ")" "" "" ""
| rex field=search mode=sed "s/=\"/(/g s/\" \) zAS\(FS_/) AS /g s/\"\s+max/ max/g s/\"\s+$//"] BY host