topic Re: Disk Usage Alerts - Linux Disk Monitoring in Alerting

Disk Usage Alerts - Linux Disk Monitoring

travistrp — Wed, 30 Jun 2010 20:56:43 GMT

Utilizing the Splunk forwarder off of our SANS box we can see disk usage across all volumes that are attached. Now that we have the data does anybody have a recommended way of alerting on a mount when it drops below x usage?

Right now I would like to setup some sort of notification / email alert to send me an email when /mnt/sansmount drops below 2.0T available disk space or 90% UsePct but for some reason I can't seem to get it working.

Has anyone done this or have any ideas?

Thanks again to everyone

Filesystem Type Size Used Avail UsePct MountedOn /dev/sda5
ext3 4.9G 1.4G 3.3G 31% / /dev/sda8
ext3 164G 96G 60G 62% /usr /dev/sda7
ext3 2.0G 763M 1.1G 41% /opt /dev/sda6
ext3 3.9G 73M 3.7G 2% /tmp /dev/sda3
ext3 4.9G 380M 4.3G 9% /var /dev/sda1
ext3 2.0G 41M 1.8G 3% /boot /dev/cvfsctl1_sansmount cvfs 18T 15T 2.8T 84% /mnt/sansmount

Re: Disk Usage Alerts - Linux Disk Monitoring

ziegfried — Wed, 30 Jun 2010 21:54:58 GMT

I'm using the following search for my alerting purposes:

sourcetype=df | multikv | dedup host,Filesystem | rex field=UsePct "(?<usage>\d+)" | where usage>90 | eval _raw="Filesystem "+Filesystem+" (mount point "+MountedOn+") on host "+host+" is "+UsePct+" full!" | fields - *

Explaination in detail:

sourcetype=df - Lists all df Events
| multikv - Splits the table-formated event into an event for each line
| dedup host,Filesystem - only keep the latest event per host and filesystem
| rex field=UsePct "(?<usage>\d+)" - Extracts a clean number form the UsePct field (ie. without the %-character) into the 'usage' field
| where usage>90 - Filters events by the given threashold
| eval _raw="Filesystem "+Filesystem+" (mount point "+MountedOn+") on host "+host+" is "+UsePct+" full!" - Overwrites the event text with a something that is better readable. This will be in the email, if the option include results is enabled
| fields - * Remove all fields

If you want to restict the alert/search to some specific mountpoints, then you would need to add another filter to your search, like:

sourcetype=df | multikv | dedup host,Filesystem | search MountedOn="/mnt/sansmount" | rex field=UsePct "(?<usage>\d+)" | where usage>90 | eval _raw="Filesystem "+Filesystem+" (mount point "+MountedOn+") on host "+host+" is "+UsePct+" full!" | fields - *

For the alerting, I've created a saved search which runs every day for the last 24 hours (-24h) and sends an email if number of events is greater than 0.

Re: Disk Usage Alerts - Linux Disk Monitoring

Lowell — Thu, 01 Jul 2010 00:48:13 GMT

Shouldn't your "| fields -" be "| fields - *`" (space between "-" and "")

Re: Disk Usage Alerts - Linux Disk Monitoring

ziegfried — Thu, 01 Jul 2010 04:32:33 GMT

Yes, you're right. I've corrected it in the post. Using "field -*" actually works as well, it would be an include of all fields that start with a "-" character. I wasn't my intention, though 😉

Re: Disk Usage Alerts - Linux Disk Monitoring

vinay_uim — Sat, 20 Aug 2016 05:25:46 GMT

Hi,

I am trying to setup disk space alert for few servers but I am not getting any results with sourcetype="df".

As I am very new splunk user, could you please let me know is there any other source type available for disk space or how can I set up alert on those servers where sourcetype=df not available?

Thanks in advance.