Getting Data In

disk usage from the monitoring console does NOT match the reality


hi fellas

I am experiencing an unexpected behavior when trying to monitor disk usage. Hopefully you can explain why this is happening

In the built-in dashboard "Resource Usage: Instance" in the DMC I am checking the disk usage of my indexers and then I am comparing those figures with the output of the df command.

This is what I get:

In the DMC:
host A Disk Usage: 2768.56 / 8400.31 (GB)
host B Disk Usage: 3429.27 / 8400.31 (GB)

While "df -h" gives me these:
host A Disk Usage: 2.7 T / 8 T
host B Disk Usage: 2.6 T / 6.4 T

As you can see, the numbers especially for host B, do not match at all.

any ideas about what must be happening???

Tags (4)

Path Finder

So no conclusion on this subject?

I have seen this type of mismatch as well. Here is an example:


-bash-4.2$ df /data3
Filesystem                    1K-blocks        Used  Available Use% Mounted on
/dev/mapper/data3vg-data3lv 12882810880 11133090120 1749720760  87% /data3



From Monitoring Console, Resource Usage:Machine

Mount Point File System Type Disk Usage (GB)

/data3 xfs 9746.51 / 12286.01


It is over 1TB difference (around 10%). This filesystem only contains cold indexes. Nothing else.


Monitoring Console page Resource Usage:Machine Disk Usage, uses REST /services/server/status/partitions-space and this sometimes give incorrect numbers.

The REST call used in Volume Details /services/data/index-volumes gives however correct figures.

0 Karma


I'm experiencing same issue;.

0 Karma

Ultra Champion

looks like the issue was in where the df -h command was executed
see comments above
Splunk is accurate DMC is great!

0 Karma


hi @adonio, the nix app is giving me the same results as the df -h command. The DMC gives a different value

0 Karma

Ultra Champion

check the mount point
what the nix app says on disk A? does it match the DMC?

0 Karma


the mount points match on both apps, the disk usage does not

0 Karma

Super Champion

yes. there is an issue.

If this helps, give a like below.
0 Karma


up! ^^^^^^

0 Karma

Ultra Champion

hello @asimagu,
looks like host A is accurate.
this is the search that powers the DMC dashboard:

| rest splunk_server=* /services/server/status/partitions-space
| eval free = if(isnotnull(available), available, free)
| eval usage = round((capacity - free) / 1024, 2)
| eval capacity = round(capacity / 1024, 2)
| eval compare_usage = usage." / ".capacity
| eval pct_usage = round(usage / capacity * 100, 2)
| stats first(fs_type) as fs_type first(compare_usage) AS compare_usage first(pct_usage) as pct_usage by mount_point

| rename mount_point as "Mount Point", fs_type as "File System Type", compare_usage as "Disk Usage (GB)", pct_usage as "Disk Usage (%)"

can you verify the df -h command is executed in the right place on file system?

0 Karma


hi @adonio
thanks for your help. I always thought that it didn´t matter where the df command was run.
it gives me the same result no matter where I am in the fylesystem....

0 Karma
Get Updates on the Splunk Community!

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...

Security Highlights | January 2023 Newsletter

January 2023 Splunk Security Essentials (SSE) 3.7.0 ReleaseThe free Splunk Security Essentials (SSE) 3.7.0 app ...

Platform Highlights | January 2023 Newsletter

 January 2023Peace on Earth and Peace of Mind With Business ResilienceAll organizations can start the new year ...