I am experiencing an unexpected behavior when trying to monitor disk usage. Hopefully you can explain why this is happening
In the built-in dashboard "Resource Usage: Instance" in the DMC I am checking the disk usage of my indexers and then I am comparing those figures with the output of the df command.
This is what I get:
In the DMC:
host A Disk Usage: 2768.56 / 8400.31 (GB)
host B Disk Usage: 3429.27 / 8400.31 (GB)
While "df -h" gives me these:
host A Disk Usage: 2.7 T / 8 T
host B Disk Usage: 2.6 T / 6.4 T
As you can see, the numbers especially for host B, do not match at all.
any ideas about what must be happening???
So no conclusion on this subject?
I have seen this type of mismatch as well. Here is an example:
-bash-4.2$ df /data3 Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/data3vg-data3lv 12882810880 11133090120 1749720760 87% /data3
From Monitoring Console, Resource Usage:Machine
Mount Point File System Type Disk Usage (GB)
/data3 xfs 9746.51 / 12286.01
It is over 1TB difference (around 10%). This filesystem only contains cold indexes. Nothing else.
Monitoring Console page Resource Usage:Machine Disk Usage, uses REST /services/server/status/partitions-space and this sometimes give incorrect numbers.
The REST call used in Volume Details /services/data/index-volumes gives however correct figures.
looks like host A is accurate.
this is the search that powers the DMC dashboard:
| rest splunk_server=* /services/server/status/partitions-space
| eval free = if(isnotnull(available), available, free)
| eval usage = round((capacity - free) / 1024, 2)
| eval capacity = round(capacity / 1024, 2)
| eval compare_usage = usage." / ".capacity
| eval pct_usage = round(usage / capacity * 100, 2)
| stats first(fs_type) as fs_type first(compare_usage) AS compare_usage first(pct_usage) as pct_usage by mount_point
| rename mount_point as "Mount Point", fs_type as "File System Type", compare_usage as "Disk Usage (GB)", pct_usage as "Disk Usage (%)"
can you verify the df -h command is executed in the right place on file system?