Monitoring Splunk

Query to setup alert if the diskspace goes over 70%?

shakeel253
Explorer

Below are the Host and Source type, I am trying to setup an alert if the diskspace goes over 70%. can some help?

host=tableau sourcetype="Perfmon:Free Disk Space"

0 Karma
1 Solution

woodcock
Esteemed Legend

Like this:

index=YouShouldAlwaysSpecifyAnIndex sourcetype="Perfmon:Free Disk Space" counter="% Free Space" host=tableaufqt
| chart avg(Value) AS PctFree BY host
| eval PctUsed = 100 - PctFree
| search PctUsed > 70

Then save this as an alert with a trigger of Number of events and is greater than 0

View solution in original post

woodcock
Esteemed Legend

Like this:

index=YouShouldAlwaysSpecifyAnIndex sourcetype="Perfmon:Free Disk Space" counter="% Free Space" host=tableaufqt
| chart avg(Value) AS PctFree BY host
| eval PctUsed = 100 - PctFree
| search PctUsed > 70

Then save this as an alert with a trigger of Number of events and is greater than 0

shakeel253
Explorer

hey Woodcock, thank you for the above mentioned query, when i ran this query, i did not get a result. The reason i didnt get a result is because hard disk is below 70. To check if the query work, i changed the percentage to 20%, i shouldve gotten something but no result to display.

Is it something that needs to be changed within the query or am i doing something wrong

0 Karma

woodcock
Esteemed Legend

If you get rid of the last line, do you get any results there?

0 Karma

shakeel253
Explorer

when i take the line 4 out, i still do not see any results

index=YouShouldAlwaysSpecifyAnIndex sourcetype="Perfmon:Free Disk Space" counter="% Free Space" host=tableaufqt host=TABLEAU
| timechart avg(Value) AS PctFree BY host
| eval PctUsed = 100 - PctFree

0 Karma

woodcock
Esteemed Legend

You have to substitute in your "real" stuff for my "fake" placeholders stuff (e.g. YouShouldAlwaysSpecifyAnIndex) and anything else that we guessed/assumed.

0 Karma

shakeel253
Explorer

Using below query,
index=* sourcetype="Perfmon:Free Disk Space" counter="% Free Space" host=tableaufqt
| timechart avg(Value) AS PctFree BY host
| eval PctUsed = 100 - PctFree

i got the diskspace

2017-09-11 12:30:00 48.86669237574543

now what should i add to the query that if it goes pass 70%, it sents out an alert?

0 Karma

woodcock
Esteemed Legend

First, change timechart to chart and then add back in the last line that we took out for debugging. Go back to the original answer now that you have made or fake-to-real substitutions.

0 Karma

woodcock
Esteemed Legend

And "*" does not count for best practices. Use the correct Index value.

0 Karma

shakeel253
Explorer

the query worked, thankyou for your help

0 Karma

richgalloway
SplunkTrust
SplunkTrust

You need more than just host and sourcetype. You also need a field showing how much diskspace is in use. Do you have one of those? If so, a sample query might look like this.

host=tableau sourcetype="Perfmon:Free Disk Space" spaceUsed=* | where spaceUsed>70 | table host spaceUsed

Once you have the query returning the desired results, schedule it to run at an appropriate interval (hourly, perhaps) and send an alert if the number of results is not zero.

---
If this reply helps you, Karma would be appreciated.
0 Karma

shakeel253
Explorer

hello Rich,

I am not getting results with the above mentioned query. One possible answer is that i do not see spaceused field, nor do i have df.

This query gives me a result
host=tableaufqt sourcetype="Perfmon:Free Disk Space" counter="% Free Space" | timechart avg(Value) as PercentFree by host

2017-09-10 14:30:00 48.89175133278226.

how can i use this query to setup alert when diskspace goes pass 70%

0 Karma

richgalloway
SplunkTrust
SplunkTrust

The spaceUsed field in my query is a pseudo-field. You need to replace it with the correct field from your data. Sorry for not mentioning that in my answer.
Searching for 'index=host=tableau sourcetype="Perfmon:Free Disk Space"' will return the available fields. If you don't see anything appropriate then you may need to extract additional fields.

---
If this reply helps you, Karma would be appreciated.
0 Karma

shakeel253
Explorer

Richgalloway i have added all the field but still not getting a result with the above mentioned query. Most probably because i do not see a spaceused field. Do you know whats missing?

date_hour   24  81.8%   Number
date_mday   4   81.8%   Number
date_minute 60  81.8%   Number
date_month  1   81.8%   String
date_second 60  81.79%  Number
date_wday   4   81.8%   String
date_year   1   81.8%   Number
date_zone   3   81.8%   Number
eventtype   4   94.36%  String
host    2   100%    String
index   1   100%    String
linecount   27  100%    Number
punct   >100    100%    String
source  >100    100%    String
sourcetype  55  100%    String
splunk_server   1   100%    String
timeendpos  34  81.8%   Number
timestartpos    16  81.8%   Number
unix_category   1   100%    String
unix_group  1   100%    String
0 Karma

niketn
Legend

I think I had answered something on similar lines in one of your previous questions: https://answers.splunk.com/answers/568907/visual-chart-for-how-much-free-disk-space-is-avail.html#an.... Please try the following query:

host=tableau sourcetype="Perfmon:Free Disk Space" object="LogicalDisk" counter="% Free Space" 
| head 1
| eval Used_Percent=round(100-Value,1)
| search Used_Percent>70
____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

shakeel253
Explorer

Hey NiketNilay , your previous query was on point on giving me the visual dashboard that i was looking for, but now i have to setup an alert if the diskspace goes pass 70% or above. The above mentioned query does not give me any results

0 Karma

niketn
Legend

What is the result you get when you run the following query?

host=tableau sourcetype="Perfmon:Free Disk Space" object="LogicalDisk" counter="% Free Space" 
| head 1

If you do not get any results, then please provide the correct base search as @woodcock has mentioned you should also include index name in your search. If Splunk admins have not set a default index and you are not allowed to search without specifying the index, your query itself might not work.

You can directly set alert on % Free Space for alert also i.e. alert for free space less than 30%

 host=tableau sourcetype="Perfmon:Free Disk Space" object="LogicalDisk" counter="% Free Space" 
| head 1
| search Value<30
____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

shakeel253
Explorer

hey Niket,
i got below results when i ran this query
host=tableau sourcetype="Perfmon:Free Disk Space" object="LogicalDisk" counter="% Free Space"
| head 1

9/11/17
5:34:49.000 PM

09/11/2017 13:34:49.727 -0400
collection="Free Disk Space"
object=LogicalDisk
counter="% Free Space"
instance=_Total
Value=80.31195185254278
host = TABLEAU index = main linecount = 6 source = Perfmon:Free Disk Space splunk_server = ip-xx-xxx-x-xxx unix_category = all_hosts unix_group = default

0 Karma

shakeel253
Explorer

when i run this query
host=tableau sourcetype="Perfmon:Free Disk Space" object="LogicalDisk" counter="% Free Space"
| head 1

i get

9/11/17
4:34:49.000 PM

09/11/2017 12:34:49.728 -0400
collection="Free Disk Space"
object=LogicalDisk
counter="% Free Space"
instance=_Total
Value=80.31558269365843
Collapse
host = TABLEAU source = Perfmon:Free Disk Space sourcetype = Perfmon:Free Disk Space

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...