Alerting

Disk Usage Alerts - Linux Disk Monitoring

travistrp
Explorer

Utilizing the Splunk forwarder off of our SANS box we can see disk usage across all volumes that are attached. Now that we have the data does anybody have a recommended way of alerting on a mount when it drops below x usage?

Right now I would like to setup some sort of notification / email alert to send me an email when /mnt/sansmount drops below 2.0T available disk space or 90% UsePct but for some reason I can't seem to get it working.

Has anyone done this or have any ideas?

Thanks again to everyone

Filesystem Type Size Used Avail UsePct MountedOn /dev/sda5
ext3 4.9G 1.4G 3.3G 31% / /dev/sda8
ext3 164G 96G 60G 62% /usr /dev/sda7
ext3 2.0G 763M 1.1G 41% /opt /dev/sda6
ext3 3.9G 73M 3.7G 2% /tmp /dev/sda3
ext3 4.9G 380M 4.3G 9% /var /dev/sda1
ext3 2.0G 41M 1.8G 3% /boot /dev/cvfsctl1_sansmount cvfs 18T 15T 2.8T 84% /mnt/sansmount

Tags (2)
1 Solution

ziegfried
Influencer

I'm using the following search for my alerting purposes:

sourcetype=df | multikv | dedup host,Filesystem | rex field=UsePct "(?<usage>\d+)" | where usage>90 | eval _raw="Filesystem "+Filesystem+" (mount point "+MountedOn+") on host "+host+" is "+UsePct+" full!" | fields - *

Explaination in detail:

  • sourcetype=df - Lists all df Events
  • | multikv - Splits the table-formated event into an event for each line
  • | dedup host,Filesystem - only keep the latest event per host and filesystem
  • | rex field=UsePct "(?<usage>\d+)" - Extracts a clean number form the UsePct field (ie. without the %-character) into the 'usage' field
  • | where usage>90 - Filters events by the given threashold
  • | eval _raw="Filesystem "+Filesystem+" (mount point "+MountedOn+") on host "+host+" is "+UsePct+" full!" - Overwrites the event text with a something that is better readable. This will be in the email, if the option include results is enabled
  • | fields - * Remove all fields

If you want to restict the alert/search to some specific mountpoints, then you would need to add another filter to your search, like:

sourcetype=df | multikv | dedup host,Filesystem | search MountedOn="/mnt/sansmount" | rex field=UsePct "(?<usage>\d+)" | where usage>90 | eval _raw="Filesystem "+Filesystem+" (mount point "+MountedOn+") on host "+host+" is "+UsePct+" full!" | fields - *

For the alerting, I've created a saved search which runs every day for the last 24 hours (-24h) and sends an email if number of events is greater than 0.

View solution in original post

ziegfried
Influencer

I'm using the following search for my alerting purposes:

sourcetype=df | multikv | dedup host,Filesystem | rex field=UsePct "(?<usage>\d+)" | where usage>90 | eval _raw="Filesystem "+Filesystem+" (mount point "+MountedOn+") on host "+host+" is "+UsePct+" full!" | fields - *

Explaination in detail:

  • sourcetype=df - Lists all df Events
  • | multikv - Splits the table-formated event into an event for each line
  • | dedup host,Filesystem - only keep the latest event per host and filesystem
  • | rex field=UsePct "(?<usage>\d+)" - Extracts a clean number form the UsePct field (ie. without the %-character) into the 'usage' field
  • | where usage>90 - Filters events by the given threashold
  • | eval _raw="Filesystem "+Filesystem+" (mount point "+MountedOn+") on host "+host+" is "+UsePct+" full!" - Overwrites the event text with a something that is better readable. This will be in the email, if the option include results is enabled
  • | fields - * Remove all fields

If you want to restict the alert/search to some specific mountpoints, then you would need to add another filter to your search, like:

sourcetype=df | multikv | dedup host,Filesystem | search MountedOn="/mnt/sansmount" | rex field=UsePct "(?<usage>\d+)" | where usage>90 | eval _raw="Filesystem "+Filesystem+" (mount point "+MountedOn+") on host "+host+" is "+UsePct+" full!" | fields - *

For the alerting, I've created a saved search which runs every day for the last 24 hours (-24h) and sends an email if number of events is greater than 0.

vinay_uim
Explorer

Hi,

I am trying to setup disk space alert for few servers but I am not getting any results with sourcetype="df".

As I am very new splunk user, could you please let me know is there any other source type available for disk space or how can I set up alert on those servers where sourcetype=df not available?

Thanks in advance.

ziegfried
Influencer

Yes, you're right. I've corrected it in the post. Using "field -*" actually works as well, it would be an include of all fields that start with a "-" character. I wasn't my intention, though 😉

0 Karma

Lowell
Super Champion

Shouldn't your "| fields -" be "| fields - *`" (space between "-" and "")

Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...