Getting Data In

disk usage from the monitoring console does NOT match the reality

asimagu
Builder

hi fellas

I am experiencing an unexpected behavior when trying to monitor disk usage. Hopefully you can explain why this is happening

In the built-in dashboard "Resource Usage: Instance" in the DMC I am checking the disk usage of my indexers and then I am comparing those figures with the output of the df command.

This is what I get:

In the DMC:
host A Disk Usage: 2768.56 / 8400.31 (GB)
host B Disk Usage: 3429.27 / 8400.31 (GB)

While "df -h" gives me these:
host A Disk Usage: 2.7 T / 8 T
host B Disk Usage: 2.6 T / 6.4 T

As you can see, the numbers especially for host B, do not match at all.

any ideas about what must be happening???

Tags (4)

pellegrini
Path Finder

So no conclusion on this subject?

I have seen this type of mismatch as well. Here is an example:

 

-bash-4.2$ df /data3
Filesystem                    1K-blocks        Used  Available Use% Mounted on
/dev/mapper/data3vg-data3lv 12882810880 11133090120 1749720760  87% /data3

 

 

From Monitoring Console, Resource Usage:Machine

Mount Point File System Type Disk Usage (GB)

/data3 xfs 9746.51 / 12286.01

 

It is over 1TB difference (around 10%). This filesystem only contains cold indexes. Nothing else.

 

Monitoring Console page Resource Usage:Machine Disk Usage, uses REST /services/server/status/partitions-space and this sometimes give incorrect numbers.

The REST call used in Volume Details /services/data/index-volumes gives however correct figures.

0 Karma

nawazns5038
Builder

I'm experiencing same issue;.

0 Karma

adonio
Ultra Champion

looks like the issue was in where the df -h command was executed
see comments above
Splunk is accurate DMC is great!

0 Karma

asimagu
Builder

hi @adonio, the nix app is giving me the same results as the df -h command. The DMC gives a different value

0 Karma

adonio
Ultra Champion

check the mount point
what the nix app says on disk A? does it match the DMC?

0 Karma

asimagu
Builder

the mount points match on both apps, the disk usage does not

0 Karma

thambisetty
SplunkTrust
SplunkTrust

yes. there is an issue.

————————————
If this helps, give a like below.
0 Karma

asimagu
Builder

up! ^^^^^^

0 Karma

adonio
Ultra Champion

hello @asimagu,
looks like host A is accurate.
this is the search that powers the DMC dashboard:

| rest splunk_server=* /services/server/status/partitions-space
| eval free = if(isnotnull(available), available, free)
| eval usage = round((capacity - free) / 1024, 2)
| eval capacity = round(capacity / 1024, 2)
| eval compare_usage = usage." / ".capacity
| eval pct_usage = round(usage / capacity * 100, 2)
| stats first(fs_type) as fs_type first(compare_usage) AS compare_usage first(pct_usage) as pct_usage by mount_point

| rename mount_point as "Mount Point", fs_type as "File System Type", compare_usage as "Disk Usage (GB)", pct_usage as "Disk Usage (%)"

can you verify the df -h command is executed in the right place on file system?

0 Karma

asimagu
Builder

hi @adonio
thanks for your help. I always thought that it didn´t matter where the df command was run.
it gives me the same result no matter where I am in the fylesystem....

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...