disk usage from the monitoring console does NOT match the reality

[asimagu]

hi fellas

I am experiencing an unexpected behavior when trying to monitor disk usage. Hopefully you can explain why this is happening

In the built-in dashboard "Resource Usage: Instance" in the DMC I am checking the disk usage of my indexers and then I am comparing those figures with the output of the df command.

This is what I get:

In the DMC:
host A Disk Usage: 2768.56 / 8400.31 (GB)
host B Disk Usage: 3429.27 / 8400.31 (GB)

While "df -h" gives me these:
host A Disk Usage: 2.7 T / 8 T
host B Disk Usage: 2.6 T / 6.4 T

As you can see, the numbers especially for host B, do not match at all.

any ideas about what must be happening???

[pellegrini]

So no conclusion on this subject?

I have seen this type of mismatch as well. Here is an example:

-bash-4.2$ df /data3
Filesystem                    1K-blocks        Used  Available Use% Mounted on
/dev/mapper/data3vg-data3lv 12882810880 11133090120 1749720760  87% /data3

From Monitoring Console, Resource Usage:Machine

Mount Point File System Type Disk Usage (GB)

/data3 xfs 9746.51 / 12286.01

It is over 1TB difference (around 10%). This filesystem only contains cold indexes. Nothing else.

Monitoring Console page Resource Usage:Machine Disk Usage, uses REST /services/server/status/partitions-space and this sometimes give incorrect numbers.

The REST call used in Volume Details /services/data/index-volumes gives however correct figures.

[nawazns5038]

I'm experiencing same issue;.

[adonio]

looks like the issue was in where the df -h command was executed
see comments above
Splunk is accurate DMC is great!

[asimagu]

hi @adonio, the nix app is giving me the same results as the df -h command. The DMC gives a different value

[adonio]

check the mount point
what the nix app says on disk A? does it match the DMC?

[asimagu]

the mount points match on both apps, the disk usage does not

[thambisetty]

yes. there is an issue.

[asimagu]

up! ^^^^^^

[adonio]

hello @asimagu,
looks like host A is accurate.
this is the search that powers the DMC dashboard:

| rest splunk_server=* /services/server/status/partitions-space
| eval free = if(isnotnull(available), available, free)
| eval usage = round((capacity - free) / 1024, 2)
| eval capacity = round(capacity / 1024, 2)
| eval compare_usage = usage." / ".capacity
| eval pct_usage = round(usage / capacity * 100, 2)
| stats first(fs_type) as fs_type first(compare_usage) AS compare_usage first(pct_usage) as pct_usage by mount_point

| rename mount_point as "Mount Point", fs_type as "File System Type", compare_usage as "Disk Usage (GB)", pct_usage as "Disk Usage (%)"

can you verify the df -h command is executed in the right place on file system?

[asimagu]

hi @adonio
thanks for your help. I always thought that it didn´t matter where the df command was run.
it gives me the same result no matter where I am in the fylesystem....