Alerting

Disk Usage Alerts - Linux Disk Monitoring

travistrp
Explorer

Utilizing the Splunk forwarder off of our SANS box we can see disk usage across all volumes that are attached. Now that we have the data does anybody have a recommended way of alerting on a mount when it drops below x usage?

Right now I would like to setup some sort of notification / email alert to send me an email when /mnt/sansmount drops below 2.0T available disk space or 90% UsePct but for some reason I can't seem to get it working.

Has anyone done this or have any ideas?

Thanks again to everyone

Filesystem Type Size Used Avail UsePct MountedOn /dev/sda5
ext3 4.9G 1.4G 3.3G 31% / /dev/sda8
ext3 164G 96G 60G 62% /usr /dev/sda7
ext3 2.0G 763M 1.1G 41% /opt /dev/sda6
ext3 3.9G 73M 3.7G 2% /tmp /dev/sda3
ext3 4.9G 380M 4.3G 9% /var /dev/sda1
ext3 2.0G 41M 1.8G 3% /boot /dev/cvfsctl1_sansmount cvfs 18T 15T 2.8T 84% /mnt/sansmount

Tags (2)
1 Solution

ziegfried
Influencer

I'm using the following search for my alerting purposes:

sourcetype=df | multikv | dedup host,Filesystem | rex field=UsePct "(?<usage>\d+)" | where usage>90 | eval _raw="Filesystem "+Filesystem+" (mount point "+MountedOn+") on host "+host+" is "+UsePct+" full!" | fields - *

Explaination in detail:

  • sourcetype=df - Lists all df Events
  • | multikv - Splits the table-formated event into an event for each line
  • | dedup host,Filesystem - only keep the latest event per host and filesystem
  • | rex field=UsePct "(?<usage>\d+)" - Extracts a clean number form the UsePct field (ie. without the %-character) into the 'usage' field
  • | where usage>90 - Filters events by the given threashold
  • | eval _raw="Filesystem "+Filesystem+" (mount point "+MountedOn+") on host "+host+" is "+UsePct+" full!" - Overwrites the event text with a something that is better readable. This will be in the email, if the option include results is enabled
  • | fields - * Remove all fields

If you want to restict the alert/search to some specific mountpoints, then you would need to add another filter to your search, like:

sourcetype=df | multikv | dedup host,Filesystem | search MountedOn="/mnt/sansmount" | rex field=UsePct "(?<usage>\d+)" | where usage>90 | eval _raw="Filesystem "+Filesystem+" (mount point "+MountedOn+") on host "+host+" is "+UsePct+" full!" | fields - *

For the alerting, I've created a saved search which runs every day for the last 24 hours (-24h) and sends an email if number of events is greater than 0.

View solution in original post

ziegfried
Influencer

I'm using the following search for my alerting purposes:

sourcetype=df | multikv | dedup host,Filesystem | rex field=UsePct "(?<usage>\d+)" | where usage>90 | eval _raw="Filesystem "+Filesystem+" (mount point "+MountedOn+") on host "+host+" is "+UsePct+" full!" | fields - *

Explaination in detail:

  • sourcetype=df - Lists all df Events
  • | multikv - Splits the table-formated event into an event for each line
  • | dedup host,Filesystem - only keep the latest event per host and filesystem
  • | rex field=UsePct "(?<usage>\d+)" - Extracts a clean number form the UsePct field (ie. without the %-character) into the 'usage' field
  • | where usage>90 - Filters events by the given threashold
  • | eval _raw="Filesystem "+Filesystem+" (mount point "+MountedOn+") on host "+host+" is "+UsePct+" full!" - Overwrites the event text with a something that is better readable. This will be in the email, if the option include results is enabled
  • | fields - * Remove all fields

If you want to restict the alert/search to some specific mountpoints, then you would need to add another filter to your search, like:

sourcetype=df | multikv | dedup host,Filesystem | search MountedOn="/mnt/sansmount" | rex field=UsePct "(?<usage>\d+)" | where usage>90 | eval _raw="Filesystem "+Filesystem+" (mount point "+MountedOn+") on host "+host+" is "+UsePct+" full!" | fields - *

For the alerting, I've created a saved search which runs every day for the last 24 hours (-24h) and sends an email if number of events is greater than 0.

vinay_uim
Explorer

Hi,

I am trying to setup disk space alert for few servers but I am not getting any results with sourcetype="df".

As I am very new splunk user, could you please let me know is there any other source type available for disk space or how can I set up alert on those servers where sourcetype=df not available?

Thanks in advance.

ziegfried
Influencer

Yes, you're right. I've corrected it in the post. Using "field -*" actually works as well, it would be an include of all fields that start with a "-" character. I wasn't my intention, though 😉

0 Karma

Lowell
Super Champion

Shouldn't your "| fields -" be "| fields - *`" (space between "-" and "")

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...