Monitoring Splunk

Alert when disk is full

silvermail
Path Finder

Hello all,

Is it possible to ask Splunk to send an email notification when the disk falls below the MinFreeSpace mark?

I understand that these activities will be logged in splunkd.log, so I can perform a search like:

“index="_internal" source="*splunkd.log" Disk is Full

And it will return me something like

"“06-17-2010 20:44:49.581 ERROR DiskMon - Disk is full on partition C:\Program Files\Splunk\var\lib\splunk\audit\db, pausing process, will resume when free disk space rises above : 2000MB”"

This seems simple for me to create a Scheduled Search + Alerting.

But the problem is when the disk is below the MinFreeSpace mark, Splunk will stop searches - so I assume my scheduled search with email alerting will also cease to work. In this case, I am unable to ask Splunk to send me the email notification, am I correct?

If so, is there a better way for this?

Thanks.

Tags (1)
1 Solution

Lowell
Super Champion

If both your dispatch folder and all of your indexes are on the same partition then yes, you could easily run into the situation where you can not report the situation because splunk cannot index or search on that information. So a pro-active approach like you are thinking is a wise approach. Fortunately, both the windows and unix apps provided by splunk have the ability to capture drive space information. You may have to enable the monitoring of these resources.

With the windows app, you should be able to do a search like this:

sourcetype="WMI:FreeDiskSpace" host=splunk.server.name Name="C:" FreeMegabytes<2048

You could also use the PercentFreeSpace if you prefer that over an explicit megabyte limit.

With the Unix app, you would be looking for index=os sourcetype=df and search is a bit more complicated due to the way the even is indexed. (Here is a search I have setup for one of my servers.)

 index="os" sourcetype="df" host=splunk.server.name | multikv fields Filesystem, UsePct, Avail | eval avail_kb=Avail | convert memk(avail_kb) | search /dev/sda1 avail_kb<2097152 | fields + _time, Filesystem, UsePct, Avail

Both of these examples only look at a single drive letter or unix device (although you could use a mount name too) and they assume a limit of 2G, which you should obviously ensure that you indexing/searching limit is set lower than that or this alert may also fail to work. 😉

View solution in original post

Lowell
Super Champion

If both your dispatch folder and all of your indexes are on the same partition then yes, you could easily run into the situation where you can not report the situation because splunk cannot index or search on that information. So a pro-active approach like you are thinking is a wise approach. Fortunately, both the windows and unix apps provided by splunk have the ability to capture drive space information. You may have to enable the monitoring of these resources.

With the windows app, you should be able to do a search like this:

sourcetype="WMI:FreeDiskSpace" host=splunk.server.name Name="C:" FreeMegabytes<2048

You could also use the PercentFreeSpace if you prefer that over an explicit megabyte limit.

With the Unix app, you would be looking for index=os sourcetype=df and search is a bit more complicated due to the way the even is indexed. (Here is a search I have setup for one of my servers.)

 index="os" sourcetype="df" host=splunk.server.name | multikv fields Filesystem, UsePct, Avail | eval avail_kb=Avail | convert memk(avail_kb) | search /dev/sda1 avail_kb<2097152 | fields + _time, Filesystem, UsePct, Avail

Both of these examples only look at a single drive letter or unix device (although you could use a mount name too) and they assume a limit of 2G, which you should obviously ensure that you indexing/searching limit is set lower than that or this alert may also fail to work. 😉

vinay_uim
Explorer

Hi Lowell,

I am trying to setup disk space alert for few servers but I am not getting any results with sourcetype="df".

As I am very new splunk user, could you please let me know is there any other source type available for disk space or how can I set up alert on those servers where sourcetype=df not available?

Thanks in advance.

0 Karma

silvermail
Path Finder

I'm thinking of running a script like df to check for the disk space, and then sending off the alerts accordingly if it drops below a certain threshold like 10% free. Something like:

index=os sourcetype=df | multikv fields UsePCT | search UsePct < "10%"

This is assuming Splunk is running on Unixes...if it's Windows, we will have to look for some other metrics to calculate.

Not sure if I am on the correct track...

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...