topic Re: Query to setup alert if the diskspace goes over 70%? in Monitoring Splunk

Query to setup alert if the diskspace goes over 70%?

shakeel253 — Mon, 11 Sep 2017 12:45:35 GMT

Below are the Host and Source type, I am trying to setup an alert if the diskspace goes over 70%. can some help?

host=tableau sourcetype="Perfmon:Free Disk Space"

Re: Query to setup alert if the diskspace goes over 70%?

niketn — Mon, 11 Sep 2017 12:54:40 GMT

I think I had answered something on similar lines in one of your previous questions: https://answers.splunk.com/answers/568907/visual-chart-for-how-much-free-disk-space-is-avail.html#answer-568941. Please try the following query:

host=tableau sourcetype="Perfmon:Free Disk Space" object="LogicalDisk" counter="% Free Space" 
| head 1
| eval Used_Percent=round(100-Value,1)
| search Used_Percent>70

Re: Query to setup alert if the diskspace goes over 70%?

richgalloway — Mon, 11 Sep 2017 13:07:26 GMT

You need more than just host and sourcetype. You also need a field showing how much diskspace is in use. Do you have one of those? If so, a sample query might look like this.

host=tableau sourcetype="Perfmon:Free Disk Space" spaceUsed=* | where spaceUsed>70 | table host spaceUsed

Once you have the query returning the desired results, schedule it to run at an appropriate interval (hourly, perhaps) and send an alert if the number of results is not zero.

Re: Query to setup alert if the diskspace goes over 70%?

shakeel253 — Mon, 11 Sep 2017 14:00:36 GMT

Richgalloway i have added all the field but still not getting a result with the above mentioned query. Most probably because i do not see a spaceused field. Do you know whats missing?

date_hour   24  81.8%   Number
date_mday   4   81.8%   Number
date_minute 60  81.8%   Number
date_month  1   81.8%   String
date_second 60  81.79%  Number
date_wday   4   81.8%   String
date_year   1   81.8%   Number
date_zone   3   81.8%   Number
eventtype   4   94.36%  String
host    2   100%    String
index   1   100%    String
linecount   27  100%    Number
punct   >100    100%    String
source  >100    100%    String
sourcetype  55  100%    String
splunk_server   1   100%    String
timeendpos  34  81.8%   Number
timestartpos    16  81.8%   Number
unix_category   1   100%    String
unix_group  1   100%    String

Re: Query to setup alert if the diskspace goes over 70%?

shakeel253 — Mon, 11 Sep 2017 14:08:27 GMT

Hey NiketNilay , your previous query was on point on giving me the visual dashboard that i was looking for, but now i have to setup an alert if the diskspace goes pass 70% or above. The above mentioned query does not give me any results

Re: Query to setup alert if the diskspace goes over 70%?

shakeel253 — Mon, 11 Sep 2017 14:27:58 GMT

hello Rich,

I am not getting results with the above mentioned query. One possible answer is that i do not see spaceused field, nor do i have df.

This query gives me a result
host=tableaufqt sourcetype="Perfmon:Free Disk Space" counter="% Free Space" | timechart avg(Value) as PercentFree by host

2017-09-10 14:30:00 48.89175133278226.

how can i use this query to setup alert when diskspace goes pass 70%

Re: Query to setup alert if the diskspace goes over 70%?

woodcock — Mon, 11 Sep 2017 16:12:58 GMT

Like this:

index=YouShouldAlwaysSpecifyAnIndex sourcetype="Perfmon:Free Disk Space" counter="% Free Space" host=tableaufqt
| chart avg(Value) AS PctFree BY host
| eval PctUsed = 100 - PctFree
| search PctUsed > 70

Then save this as an alert with a trigger of Number of events and is greater than 0

Re: Query to setup alert if the diskspace goes over 70%?

shakeel253 — Mon, 11 Sep 2017 16:39:58 GMT

hey Woodcock, thank you for the above mentioned query, when i ran this query, i did not get a result. The reason i didnt get a result is because hard disk is below 70. To check if the query work, i changed the percentage to 20%, i shouldve gotten something but no result to display.

Is it something that needs to be changed within the query or am i doing something wrong

Re: Query to setup alert if the diskspace goes over 70%?

niketn — Mon, 11 Sep 2017 16:54:19 GMT

What is the result you get when you run the following query?

host=tableau sourcetype="Perfmon:Free Disk Space" object="LogicalDisk" counter="% Free Space" 
| head 1

If you do not get any results, then please provide the correct base search as @woodcock has mentioned you should also include index name in your search. If Splunk admins have not set a default index and you are not allowed to search without specifying the index, your query itself might not work.

You can directly set alert on % Free Space for alert also i.e. alert for free space less than 30%

 host=tableau sourcetype="Perfmon:Free Disk Space" object="LogicalDisk" counter="% Free Space" 
| head 1
| search Value<30

Re: Query to setup alert if the diskspace goes over 70%?

shakeel253 — Mon, 11 Sep 2017 17:13:44 GMT

when i run this query
host=tableau sourcetype="Perfmon:Free Disk Space" object="LogicalDisk" counter="% Free Space"
| head 1

i get

9/11/17
4:34:49.000 PM

09/11/2017 12:34:49.728 -0400
collection="Free Disk Space"
object=LogicalDisk
counter="% Free Space"
instance=_Total
Value=80.31558269365843
Collapse
host = TABLEAU source = Perfmon:Free Disk Space sourcetype = Perfmon:Free Disk Space

Re: Query to setup alert if the diskspace goes over 70%?

shakeel253 — Tue, 29 Sep 2020 15:41:40 GMT

hey Niket,
i got below results when i ran this query
host=tableau sourcetype="Perfmon:Free Disk Space" object="LogicalDisk" counter="% Free Space"
| head 1

9/11/17
5:34:49.000 PM

09/11/2017 13:34:49.727 -0400
collection="Free Disk Space"
object=LogicalDisk
counter="% Free Space"
instance=_Total
Value=80.31195185254278
host = TABLEAU index = main linecount = 6 source = Perfmon:Free Disk Space splunk_server = ip-xx-xxx-x-xxx unix_category = all_hosts unix_group = default

Re: Query to setup alert if the diskspace goes over 70%?

woodcock — Mon, 11 Sep 2017 18:32:45 GMT

If you get rid of the last line, do you get any results there?

Re: Query to setup alert if the diskspace goes over 70%?

shakeel253 — Mon, 11 Sep 2017 18:40:34 GMT

when i take the line 4 out, i still do not see any results

index=YouShouldAlwaysSpecifyAnIndex sourcetype="Perfmon:Free Disk Space" counter="% Free Space" host=tableaufqt host=TABLEAU
| timechart avg(Value) AS PctFree BY host
| eval PctUsed = 100 - PctFree

Re: Query to setup alert if the diskspace goes over 70%?

richgalloway — Mon, 11 Sep 2017 20:47:38 GMT

The spaceUsed field in my query is a pseudo-field. You need to replace it with the correct field from your data. Sorry for not mentioning that in my answer.
Searching for 'index=host=tableau sourcetype="Perfmon:Free Disk Space"' will return the available fields. If you don't see anything appropriate then you may need to extract additional fields.

Re: Query to setup alert if the diskspace goes over 70%?

woodcock — Tue, 12 Sep 2017 00:26:40 GMT

You have to substitute in your "real" stuff for my "fake" placeholders stuff (e.g. YouShouldAlwaysSpecifyAnIndex) and anything else that we guessed/assumed.

Re: Query to setup alert if the diskspace goes over 70%?

shakeel253 — Tue, 12 Sep 2017 12:37:46 GMT

Using below query,
index=* sourcetype="Perfmon:Free Disk Space" counter="% Free Space" host=tableaufqt
| timechart avg(Value) AS PctFree BY host
| eval PctUsed = 100 - PctFree

i got the diskspace

2017-09-11 12:30:00 48.86669237574543

now what should i add to the query that if it goes pass 70%, it sents out an alert?

Re: Query to setup alert if the diskspace goes over 70%?

woodcock — Tue, 12 Sep 2017 14:25:12 GMT

First, change timechart to chart and then add back in the last line that we took out for debugging. Go back to the original answer now that you have made or fake-to-real substitutions.

Re: Query to setup alert if the diskspace goes over 70%?

woodcock — Tue, 12 Sep 2017 14:25:51 GMT

And "*" does not count for best practices. Use the correct Index value.

Re: Query to setup alert if the diskspace goes over 70%?

shakeel253 — Tue, 12 Sep 2017 16:29:02 GMT

the query worked, thankyou for your help