topic How to setup email alerts when linux servers are 80% full? in Alerting

How to setup email alerts when linux servers are 80% full?

mike000 — Tue, 09 Jun 2020 22:03:25 GMT

As always I know you will be able to answer my question.
So using this query:

index=_nix_xxxx sourcetype=df host=abdhw003 OR host=n OR host=n OR host=n or host=n MountedOn="/doc"
 | eval TotalGBytes= TotalMBytes/1024 | eval UsedGBytes=UsedMbytes/1024 |eval used_pct=100(UsedGBytes/TotalGBytes) 
 | stats max(TotalGBytes) as "MaxSize(GB) max(UsedGBytes) as "UsedSize(GB) as "percentUsed" by host, MountedOn 
 | search PercentUsed>05| Sort PercentUsed

I am able to see the space used by each server, is there a way wherein the dashboard once any server hits 80% or 90% used- the color of that server changes to red and an email is triggered to the support team that a certain server has reached 90% capacity?

Is that a query or something to be parameterized in the dashboard itself? Trying to understand Splunk, I appreciate all the help.

Thanks,
Mike

Re: How to setup email alerts when linux servers are 80% full?

woodcock — Fri, 03 Apr 2020 18:24:09 GMT

Something like this:

index="_nix_xxxx " AND sourcetype="df" 
| stats avg(PercentUsedSpace) AS avgPctUsed BY host Filesystem 
| eval threshold = case( 
   match(Filesystem, "foo"), 85, 
   ...
   true(), 90) 
| where avgPctUsed > threshold
| sort 0 - avgPctUsed

Re: How to setup email alerts when linux servers are 80% full?

mike000 — Fri, 03 Apr 2020 20:15:15 GMT

Hi woodcock, thanks for your reply. If I may ask, What will this do? Does this part just change the color or send email trigger? I cant see either in the snippet.

Re: How to setup email alerts when linux servers are 80% full?

woodcock — Fri, 03 Apr 2020 20:59:31 GMT

Just save it as an alert, add the alert action of email, done.