topic Re: disk usage from the monitoring console does NOT match the reality in Getting Data In

disk usage from the monitoring console does NOT match the reality

asimagu — Tue, 09 May 2017 14:13:19 GMT

hi fellas

I am experiencing an unexpected behavior when trying to monitor disk usage. Hopefully you can explain why this is happening

In the built-in dashboard "Resource Usage: Instance" in the DMC I am checking the disk usage of my indexers and then I am comparing those figures with the output of the df command.

This is what I get:

In the DMC:
host A Disk Usage: 2768.56 / 8400.31 (GB)
host B Disk Usage: 3429.27 / 8400.31 (GB)

While "df -h" gives me these:
host A Disk Usage: 2.7 T / 8 T
host B Disk Usage: 2.6 T / 6.4 T

As you can see, the numbers especially for host B, do not match at all.

any ideas about what must be happening???

Re: disk usage from the monitoring console does NOT match the reality

asimagu — Fri, 12 May 2017 12:43:30 GMT

up! ^^^^^^

Re: disk usage from the monitoring console does NOT match the reality

adonio — Tue, 29 Sep 2020 14:03:04 GMT

hello @asimagu,
looks like host A is accurate.
this is the search that powers the DMC dashboard:

| rest splunk_server=* /services/server/status/partitions-space
| eval free = if(isnotnull(available), available, free)
| eval usage = round((capacity - free) / 1024, 2)
| eval capacity = round(capacity / 1024, 2)
| eval compare_usage = usage." / ".capacity
| eval pct_usage = round(usage / capacity * 100, 2)
| stats first(fs_type) as fs_type first(compare_usage) AS compare_usage first(pct_usage) as pct_usage by mount_point

| rename mount_point as "Mount Point", fs_type as "File System Type", compare_usage as "Disk Usage (GB)", pct_usage as "Disk Usage (%)"

can you verify the df -h command is executed in the right place on file system?

Re: disk usage from the monitoring console does NOT match the reality

asimagu — Fri, 12 May 2017 13:03:58 GMT

hi @adonio
thanks for your help. I always thought that it didn´t matter where the df command was run.
it gives me the same result no matter where I am in the fylesystem....

Re: disk usage from the monitoring console does NOT match the reality

adonio — Fri, 12 May 2017 13:07:08 GMT

looks like the issue was in where the df -h command was executed
see comments above
Splunk is accurate DMC is great!

Re: disk usage from the monitoring console does NOT match the reality

asimagu — Fri, 12 May 2017 13:08:35 GMT

hi @adonio, the nix app is giving me the same results as the df -h command. The DMC gives a different value

Re: disk usage from the monitoring console does NOT match the reality

adonio — Fri, 12 May 2017 13:18:58 GMT

check the mount point
what the nix app says on disk A? does it match the DMC?

Re: disk usage from the monitoring console does NOT match the reality

asimagu — Fri, 12 May 2017 13:32:22 GMT

the mount points match on both apps, the disk usage does not

Re: disk usage from the monitoring console does NOT match the reality

nawazns5038 — Thu, 13 Sep 2018 01:03:10 GMT

I'm experiencing same issue;.

Re: disk usage from the monitoring console does NOT match the reality

thambisetty — Wed, 26 Sep 2018 06:07:21 GMT

yes. there is an issue.

Re: disk usage from the monitoring console does NOT match the reality

pellegrini — Tue, 19 Jan 2021 09:20:24 GMT

So no conclusion on this subject?

I have seen this type of mismatch as well. Here is an example:

-bash-4.2$ df /data3 Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/data3vg-data3lv 12882810880 11133090120 1749720760 87% /data3

From Monitoring Console, Resource Usage:Machine

Mount Point File System Type Disk Usage (GB)

/data3 xfs 9746.51 / 12286.01

It is over 1TB difference (around 10%). This filesystem only contains cold indexes. Nothing else.

Monitoring Console page Resource Usage:Machine Disk Usage, uses REST /services/server/status/partitions-space and this sometimes give incorrect numbers.

The REST call used in Volume Details /services/data/index-volumes gives however correct figures.