Hello all,
Is it possible to ask Splunk to send an email notification when the disk falls below the MinFreeSpace mark?
I understand that these activities will be logged in splunkd.log, so I can perform a search like:
“index="_internal" source="*splunkd.log" Disk is Full
And it will return me something like
"“06-17-2010 20:44:49.581 ERROR DiskMon - Disk is full on partition C:\Program Files\Splunk\var\lib\splunk\audit\db, pausing process, will resume when free disk space rises above : 2000MB”"
This seems simple for me to create a Scheduled Search + Alerting.
But the problem is when the disk is below the MinFreeSpace mark, Splunk will stop searches - so I assume my scheduled search with email alerting will also cease to work. In this case, I am unable to ask Splunk to send me the email notification, am I correct?
If so, is there a better way for this?
Thanks.
If both your dispatch
folder and all of your indexes are on the same partition then yes, you could easily run into the situation where you can not report the situation because splunk cannot index or search on that information. So a pro-active approach like you are thinking is a wise approach. Fortunately, both the windows
and unix
apps provided by splunk have the ability to capture drive space information. You may have to enable the monitoring of these resources.
With the windows app, you should be able to do a search like this:
sourcetype="WMI:FreeDiskSpace" host=splunk.server.name Name="C:" FreeMegabytes<2048
You could also use the PercentFreeSpace
if you prefer that over an explicit megabyte limit.
With the Unix app, you would be looking for index=os sourcetype=df
and search is a bit more complicated due to the way the even is indexed. (Here is a search I have setup for one of my servers.)
index="os" sourcetype="df" host=splunk.server.name | multikv fields Filesystem, UsePct, Avail | eval avail_kb=Avail | convert memk(avail_kb) | search /dev/sda1 avail_kb<2097152 | fields + _time, Filesystem, UsePct, Avail
Both of these examples only look at a single drive letter or unix device (although you could use a mount name too) and they assume a limit of 2G, which you should obviously ensure that you indexing/searching limit is set lower than that or this alert may also fail to work. 😉
If both your dispatch
folder and all of your indexes are on the same partition then yes, you could easily run into the situation where you can not report the situation because splunk cannot index or search on that information. So a pro-active approach like you are thinking is a wise approach. Fortunately, both the windows
and unix
apps provided by splunk have the ability to capture drive space information. You may have to enable the monitoring of these resources.
With the windows app, you should be able to do a search like this:
sourcetype="WMI:FreeDiskSpace" host=splunk.server.name Name="C:" FreeMegabytes<2048
You could also use the PercentFreeSpace
if you prefer that over an explicit megabyte limit.
With the Unix app, you would be looking for index=os sourcetype=df
and search is a bit more complicated due to the way the even is indexed. (Here is a search I have setup for one of my servers.)
index="os" sourcetype="df" host=splunk.server.name | multikv fields Filesystem, UsePct, Avail | eval avail_kb=Avail | convert memk(avail_kb) | search /dev/sda1 avail_kb<2097152 | fields + _time, Filesystem, UsePct, Avail
Both of these examples only look at a single drive letter or unix device (although you could use a mount name too) and they assume a limit of 2G, which you should obviously ensure that you indexing/searching limit is set lower than that or this alert may also fail to work. 😉
Hi Lowell,
I am trying to setup disk space alert for few servers but I am not getting any results with sourcetype="df".
As I am very new splunk user, could you please let me know is there any other source type available for disk space or how can I set up alert on those servers where sourcetype=df not available?
Thanks in advance.
I'm thinking of running a script like df to check for the disk space, and then sending off the alerts accordingly if it drops below a certain threshold like 10% free. Something like:
index=os sourcetype=df | multikv fields UsePCT | search UsePct < "10%"
This is assuming Splunk is running on Unixes...if it's Windows, we will have to look for some other metrics to calculate.
Not sure if I am on the correct track...