Getting Data In

disk usage from the monitoring console does NOT match the reality

asimagu
Builder

hi fellas

I am experiencing an unexpected behavior when trying to monitor disk usage. Hopefully you can explain why this is happening

In the built-in dashboard "Resource Usage: Instance" in the DMC I am checking the disk usage of my indexers and then I am comparing those figures with the output of the df command.

This is what I get:

In the DMC:
host A Disk Usage: 2768.56 / 8400.31 (GB)
host B Disk Usage: 3429.27 / 8400.31 (GB)

While "df -h" gives me these:
host A Disk Usage: 2.7 T / 8 T
host B Disk Usage: 2.6 T / 6.4 T

As you can see, the numbers especially for host B, do not match at all.

any ideas about what must be happening???

Tags (4)

pellegrini
Path Finder

So no conclusion on this subject?

I have seen this type of mismatch as well. Here is an example:

 

-bash-4.2$ df /data3
Filesystem                    1K-blocks        Used  Available Use% Mounted on
/dev/mapper/data3vg-data3lv 12882810880 11133090120 1749720760  87% /data3

 

 

From Monitoring Console, Resource Usage:Machine

Mount Point File System Type Disk Usage (GB)

/data3 xfs 9746.51 / 12286.01

 

It is over 1TB difference (around 10%). This filesystem only contains cold indexes. Nothing else.

 

Monitoring Console page Resource Usage:Machine Disk Usage, uses REST /services/server/status/partitions-space and this sometimes give incorrect numbers.

The REST call used in Volume Details /services/data/index-volumes gives however correct figures.

0 Karma

nawazns5038
Builder

I'm experiencing same issue;.

0 Karma

adonio
SplunkTrust
SplunkTrust

looks like the issue was in where the df -h command was executed
see comments above
Splunk is accurate DMC is great!

0 Karma

asimagu
Builder

hi @adonio, the nix app is giving me the same results as the df -h command. The DMC gives a different value

0 Karma

adonio
SplunkTrust
SplunkTrust

check the mount point
what the nix app says on disk A? does it match the DMC?

0 Karma

asimagu
Builder

the mount points match on both apps, the disk usage does not

0 Karma

thambisetty
Super Champion

yes. there is an issue.

————————————
If this helps, give a like below.
0 Karma

asimagu
Builder

up! ^^^^^^

0 Karma

adonio
SplunkTrust
SplunkTrust

hello @asimagu,
looks like host A is accurate.
this is the search that powers the DMC dashboard:

| rest splunk_server=* /services/server/status/partitions-space
| eval free = if(isnotnull(available), available, free)
| eval usage = round((capacity - free) / 1024, 2)
| eval capacity = round(capacity / 1024, 2)
| eval compare_usage = usage." / ".capacity
| eval pct_usage = round(usage / capacity * 100, 2)
| stats first(fs_type) as fs_type first(compare_usage) AS compare_usage first(pct_usage) as pct_usage by mount_point

| rename mount_point as "Mount Point", fs_type as "File System Type", compare_usage as "Disk Usage (GB)", pct_usage as "Disk Usage (%)"

can you verify the df -h command is executed in the right place on file system?

0 Karma

asimagu
Builder

hi @adonio
thanks for your help. I always thought that it didn´t matter where the df command was run.
it gives me the same result no matter where I am in the fylesystem....

0 Karma
Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!