Disk Usage Alerts - Linux Disk Monitoring

[travistrp]

Utilizing the Splunk forwarder off of our SANS box we can see disk usage across all volumes that are attached. Now that we have the data does anybody have a recommended way of alerting on a mount when it drops below x usage?

Right now I would like to setup some sort of notification / email alert to send me an email when /mnt/sansmount drops below 2.0T available disk space or 90% UsePct but for some reason I can't seem to get it working.

Has anyone done this or have any ideas?

Thanks again to everyone

Filesystem Type Size Used Avail UsePct MountedOn /dev/sda5
ext3 4.9G 1.4G 3.3G 31% / /dev/sda8
ext3 164G 96G 60G 62% /usr /dev/sda7
ext3 2.0G 763M 1.1G 41% /opt /dev/sda6
ext3 3.9G 73M 3.7G 2% /tmp /dev/sda3
ext3 4.9G 380M 4.3G 9% /var /dev/sda1
ext3 2.0G 41M 1.8G 3% /boot /dev/cvfsctl1_sansmount cvfs 18T 15T 2.8T 84% /mnt/sansmount

[ziegfried · Accepted solution]

I'm using the following search for my alerting purposes:

sourcetype=df | multikv | dedup host,Filesystem | rex field=UsePct "(?<usage>\d+)" | where usage>90 | eval _raw="Filesystem "+Filesystem+" (mount point "+MountedOn+") on host "+host+" is "+UsePct+" full!" | fields - *

Explaination in detail:

• sourcetype=df - Lists all df Events
• | multikv - Splits the table-formated event into an event for each line
• | dedup host,Filesystem - only keep the latest event per host and filesystem
• | rex field=UsePct "(?<usage>\d+)" - Extracts a clean number form the UsePct field (ie. without the %-character) into the 'usage' field
• | where usage>90 - Filters events by the given threashold
• | eval _raw="Filesystem "+Filesystem+" (mount point "+MountedOn+") on host "+host+" is "+UsePct+" full!" - Overwrites the event text with a something that is better readable. This will be in the email, if the option include results is enabled
• | fields - * Remove all fields

If you want to restict the alert/search to some specific mountpoints, then you would need to add another filter to your search, like:

sourcetype=df | multikv | dedup host,Filesystem | search MountedOn="/mnt/sansmount" | rex field=UsePct "(?<usage>\d+)" | where usage>90 | eval _raw="Filesystem "+Filesystem+" (mount point "+MountedOn+") on host "+host+" is "+UsePct+" full!" | fields - *

For the alerting, I've created a saved search which runs every day for the last 24 hours (-24h) and sends an email if number of events is greater than 0.

[vinay_uim]

Hi,

I am trying to setup disk space alert for few servers but I am not getting any results with sourcetype="df".

As I am very new splunk user, could you please let me know is there any other source type available for disk space or how can I set up alert on those servers where sourcetype=df not available?

Thanks in advance.

[ziegfried]

Yes, you're right. I've corrected it in the post. Using "field -*" actually works as well, it would be an include of all fields that start with a "-" character. I wasn't my intention, though 😉

[Lowell]

Shouldn't your "| fields -" be "| fields - *`" (space between "-" and "")