Hi ,
I am using following( default) query for near critical disk alert on Indexer nodes. The daily results are showing 99% where as actual disk usage is much lower. Can you help clarify. I will submit the actual support contract later.
Thanks,
Naren
| rest splunk_server_group=dmc_group_* /services/server/status/partitions-space
| eval free = if(isnotnull(available), available, free)
| eval usage = capacity - free
| eval pct_usage = floor(usage / capacity * 100)
| where pct_usage > 92
| stats first(fs_type) as fs_type first(capacity) AS capacity first(usage) AS usage first(pct_usage) AS pct_usage by splunk_server, mount_point
| eval usage = round(usage / 1024, 2)
| eval capacity = round(capacity / 1024, 2)
| rename splunk_server AS Instance mount_point as "Mount Point", fs_type as "File System Type", usage as "Usage (GB)", capacity as "Capacity (GB)", pct_usage as "Usage (%)"
Alert search Results:
Instance Mount Point File System Type Capacity (GB) Usage (GB) Usage (%)
prd-sjc-splunk-indexer-1 /opt/colddb ext4 14881.80 14239.33 95
prd-sjc-splunk-indexer-2 /opt/colddb ext4 14881.80 14523.47 97
prd-sjc-splunk-indexer-3 /opt/colddb ext4 14881.80 14664.28 98
prd-sjc-splunk-indexer-4 /opt/colddb ext4 14881.80 14845.24 99
prd-sjc-splunk-indexer-5 /opt/colddb ext4 14881.80 14612.96 98
prd-sjc-splunk-indexer-6 /opt/colddb ext4 14881.80 14744.09 99
Actual Disk space:
Processing on prd-sjc-splunk-indexer-2
/dev/mapper/hot-hot 10403135808 7814205760 2064642256 80% /opt/splunk
/dev/mapper/cold-cold 15604702004 10252296568 4565973644 70% /opt/colddb
Processing on prd-sjc-splunk-indexer-1:
/dev/mapper/hot-hot 10403135808 7956960136 1921887880 81% /opt/splunk
/dev/mapper/cold-cold 15604702004 9749420004 5068850208 66% /opt/colddb
Processing on prd-sjc-splunk-indexer-5
/dev/xvdg 10403139904 7912240516 1966611388 81% /opt/splunk
/dev/mapper/colddb-colddb 15604697908 9752163196 5066103124 66% /opt/colddb
Processing on prd-sjc-splunk-indexer-3:
/dev/mapper/hot-hot 10403135808 7865249624 2013598392 80% /opt/splunk
/dev/mapper/cold-cold 15604702004 9997688028 4820582184 68% /opt/colddb
Processing on prd-sjc-splunk-indexer-4
/dev/mapper/cold-colddb 15604697908 10681093532 4137236996 73% /opt/colddb
/dev/mapper/hot-hotdb 10403135808 7779513904 2099334112 79% /opt/splunk
Processing on prd-sjc-splunk-indexer-6
/dev/xvdg 10321219904 7766226312 2030705592 80% /opt/splunk
/dev/mapper/colddb-colddb 15604697908 10138947556 4679318764 69% /opt/colddb
Processing on prd-sjc-splunk-indexer-7
/dev/xvdg 10403139904 7783278992 2095572912 79% /opt/splunk
/dev/xvdh 17111506844 1754748380 14497765008 11% /opt/colddb
updated to mark code and data as code
Splunk reports the available space as: Total Size - (Used + Reserved)
By default, ext3/4 filesystem dedicates 5% of total space to be reserved.
So if your partition total size is 1TB and the partition is completely empty, Splunk (DMC) will report it as 50GB out of 1000GB used.
The first thing I note is that the issues are only on your colddb.
Run this to see if the available vs free numbers look different on the cold vs the hot mountpoints.
| rest splunk_server_group=dmc_group_* /services/server/status/partitions-space
| eval usage = capacity - coalesce(available,free)
| eval pct_usage = floor(100*usage/capacity)
| where pct_usage > 80
| stats first(fs_type) as fs_type first(capacity) AS capacity first(usage) AS usage first(pct_usage) AS pct_usage
first(free) as free first(available) as available by splunk_server, mount_point
| eval usage = round(usage / 1024, 2)
| eval capacity = round(capacity / 1024, 2)
| eval free = round(free / 1024, 2)
| eval available = round(available / 1024, 2)
| rename splunk_server AS Instance mount_point as "Mount Point", fs_type as "File System Type", usage as "Usage (GB)", capacity as "Capacity (GB)", pct_usage as "Usage (%)", free as "free (GB)", available as "available(GB)"
Thanks . Results are same as before.
Is there any other query we can run?. I am trying to access the license info from my dmc . Can you point me to a doc to establish support account. So that I can convert this to support ticket.