Alerting

Disk Usage Alerts - Linux Disk Monitoring

travistrp
Explorer

Utilizing the Splunk forwarder off of our SANS box we can see disk usage across all volumes that are attached. Now that we have the data does anybody have a recommended way of alerting on a mount when it drops below x usage?

Right now I would like to setup some sort of notification / email alert to send me an email when /mnt/sansmount drops below 2.0T available disk space or 90% UsePct but for some reason I can't seem to get it working.

Has anyone done this or have any ideas?

Thanks again to everyone

Filesystem Type Size Used Avail UsePct MountedOn /dev/sda5
ext3 4.9G 1.4G 3.3G 31% / /dev/sda8
ext3 164G 96G 60G 62% /usr /dev/sda7
ext3 2.0G 763M 1.1G 41% /opt /dev/sda6
ext3 3.9G 73M 3.7G 2% /tmp /dev/sda3
ext3 4.9G 380M 4.3G 9% /var /dev/sda1
ext3 2.0G 41M 1.8G 3% /boot /dev/cvfsctl1_sansmount cvfs 18T 15T 2.8T 84% /mnt/sansmount

Tags (2)
1 Solution

ziegfried
Influencer

I'm using the following search for my alerting purposes:

sourcetype=df | multikv | dedup host,Filesystem | rex field=UsePct "(?<usage>\d+)" | where usage>90 | eval _raw="Filesystem "+Filesystem+" (mount point "+MountedOn+") on host "+host+" is "+UsePct+" full!" | fields - *

Explaination in detail:

  • sourcetype=df - Lists all df Events
  • | multikv - Splits the table-formated event into an event for each line
  • | dedup host,Filesystem - only keep the latest event per host and filesystem
  • | rex field=UsePct "(?<usage>\d+)" - Extracts a clean number form the UsePct field (ie. without the %-character) into the 'usage' field
  • | where usage>90 - Filters events by the given threashold
  • | eval _raw="Filesystem "+Filesystem+" (mount point "+MountedOn+") on host "+host+" is "+UsePct+" full!" - Overwrites the event text with a something that is better readable. This will be in the email, if the option include results is enabled
  • | fields - * Remove all fields

If you want to restict the alert/search to some specific mountpoints, then you would need to add another filter to your search, like:

sourcetype=df | multikv | dedup host,Filesystem | search MountedOn="/mnt/sansmount" | rex field=UsePct "(?<usage>\d+)" | where usage>90 | eval _raw="Filesystem "+Filesystem+" (mount point "+MountedOn+") on host "+host+" is "+UsePct+" full!" | fields - *

For the alerting, I've created a saved search which runs every day for the last 24 hours (-24h) and sends an email if number of events is greater than 0.

View solution in original post

ziegfried
Influencer

I'm using the following search for my alerting purposes:

sourcetype=df | multikv | dedup host,Filesystem | rex field=UsePct "(?<usage>\d+)" | where usage>90 | eval _raw="Filesystem "+Filesystem+" (mount point "+MountedOn+") on host "+host+" is "+UsePct+" full!" | fields - *

Explaination in detail:

  • sourcetype=df - Lists all df Events
  • | multikv - Splits the table-formated event into an event for each line
  • | dedup host,Filesystem - only keep the latest event per host and filesystem
  • | rex field=UsePct "(?<usage>\d+)" - Extracts a clean number form the UsePct field (ie. without the %-character) into the 'usage' field
  • | where usage>90 - Filters events by the given threashold
  • | eval _raw="Filesystem "+Filesystem+" (mount point "+MountedOn+") on host "+host+" is "+UsePct+" full!" - Overwrites the event text with a something that is better readable. This will be in the email, if the option include results is enabled
  • | fields - * Remove all fields

If you want to restict the alert/search to some specific mountpoints, then you would need to add another filter to your search, like:

sourcetype=df | multikv | dedup host,Filesystem | search MountedOn="/mnt/sansmount" | rex field=UsePct "(?<usage>\d+)" | where usage>90 | eval _raw="Filesystem "+Filesystem+" (mount point "+MountedOn+") on host "+host+" is "+UsePct+" full!" | fields - *

For the alerting, I've created a saved search which runs every day for the last 24 hours (-24h) and sends an email if number of events is greater than 0.

vinay_uim
Explorer

Hi,

I am trying to setup disk space alert for few servers but I am not getting any results with sourcetype="df".

As I am very new splunk user, could you please let me know is there any other source type available for disk space or how can I set up alert on those servers where sourcetype=df not available?

Thanks in advance.

ziegfried
Influencer

Yes, you're right. I've corrected it in the post. Using "field -*" actually works as well, it would be an include of all fields that start with a "-" character. I wasn't my intention, though 😉

0 Karma

Lowell
Super Champion

Shouldn't your "| fields -" be "| fields - *`" (space between "-" and "")

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...