Archive
Highlighted

Is it possible to monitor file system /opt disk space and trigger an alert when it reached above 75 % of the total size of the file system.

Motivator

Hi All, We have an requirement to monitor the disk space of the file system /opt and /splogs in all the splunk servers and trigger an alert if the file system reached 75 % of the total disk space.

Currently we are doing it manually by login into all the splunk servers and executing UNIX command to find out the space and clearing the top 50 directories which is consuming more space.

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_splunk01-optvol
39G 29G 8.3G 78% /opt

df -h /splogs
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vgsplunk01san-splogs
5.3T 4.0T 1.1T 79% /splogs

Kindly guide me in setting up this monitoring from Splunk.

thanks in advance.

Tags (1)
0 Karma
Highlighted

Re: Is it possible to monitor file system /opt disk space and trigger an alert when it reached above 75 % of the total size of the file system.

Legend

See in Distributed Management Console alert "DMC Alert - Near Critical Disk Usage"
You can access it at the URL
https://xxx.xxx.xxx.xxx/splunk/en-EN/app/splunk_management_console/alerts

Bye.
Giuseppe

0 Karma
Highlighted

Re: Is it possible to monitor file system /opt disk space and trigger an alert when it reached above 75 % of the total size of the file system.

Motivator

thanks Giuseppe, but we are using splunk 6.0.3 version and I did not find the DMC console, so is there a way to write a query to fetch this information (disk space) from all splunk servers.

thanks in advance.

0 Karma
Highlighted

Re: Is it possible to monitor file system /opt disk space and trigger an alert when it reached above 75 % of the total size of the file system.

Legend

try the search

| rest splunk_server_group=dmc_group_* /services/server/status/partitions-space 
| eval free = if(isnotnull(available), available, free) 
| eval usage = capacity - free 
| eval pct_usage = floor(usage / capacity * 100) 
| where pct_usage > 80 
| stats first(fs_type) as fs_type first(capacity) AS capacity first(usage) AS usage first(pct_usage) AS pct_usage by splunk_server, mount_point 
| eval usage = round(usage / 1024, 2) 
| eval capacity = round(capacity / 1024, 2) 
| rename splunk_server AS Instance mount_point as "Mount Point", fs_type as "File System Type", usage as "Usage (GB)", capacity as "Capacity (GB)", pct_usage as "Usage (%)"

Bye.
Giuseppe

0 Karma
Highlighted

Re: Is it possible to monitor file system /opt disk space and trigger an alert when it reached above 75 % of the total size of the file system.

Motivator

Hi Giuseppe, I ran the above query and got no result. Kept the duration for an hour / day, but no luck. Kindly guide me in getting the required details. thanks in advance.

0 Karma
Highlighted

Re: Is it possible to monitor file system /opt disk space and trigger an alert when it reached above 75 % of the total size of the file system.

Legend

modify in this way

| rest  /services/server/status/partitions-space 
 | eval free = if(isnotnull(available), available, free) 
 | eval usage = capacity - free 
 | eval pct_usage = floor(usage / capacity * 100) 
 | where pct_usage > 80 
 | stats first(fs_type) as fs_type first(capacity) AS capacity first(usage) AS usage first(pct_usage) AS pct_usage by splunk_server, mount_point 
 | eval usage = round(usage / 1024, 2) 
 | eval capacity = round(capacity / 1024, 2) 
 | rename splunk_server AS Instance mount_point as "Mount Point", fs_type as "File System Type", usage as "Usage (GB)", capacity as "Capacity (GB)", pct_usage as "Usage (%)"

Bye.
Giuseppe

0 Karma
Highlighted

Re: Is it possible to monitor file system /opt disk space and trigger an alert when it reached above 75 % of the total size of the file system.

Motivator

thanks Giuseppe for providing the query but when I executed the above query I am getting result only for splunk component related to Index instances, not for the other components like search head, Heavy Forwarder. Is it possible to get those details in the same query.

thanks in advance.

0 Karma
Highlighted

Re: Is it possible to monitor file system /opt disk space and trigger an alert when it reached above 75 % of the total size of the file system.

SplunkTrust
SplunkTrust

These REST Api endpoints are for Instrospection and they were introduced in Splunk 6.2 onwards. Rest query to this enpoint will not work in lower versions.

0 Karma
Highlighted

Re: Is it possible to monitor file system /opt disk space and trigger an alert when it reached above 75 % of the total size of the file system.

SplunkTrust
SplunkTrust

Hi,

The Nmon performance application can do this for you. (it does this by default for all file systems already)
This is part of the JFSFILE, for file systems percent of utilisation.

https://splunkbase.splunk.com/app/1753/

Guilhem

0 Karma
Highlighted

Re: Is it possible to monitor file system /opt disk space and trigger an alert when it reached above 75 % of the total size of the file system.

Motivator

thanks Guilhem, we are using 6.0.3 version so is there a way to get this information by executing splunk query.

thanks in advance.

0 Karma