Hi ,
I am using following( default) query for near critical disk alert on Indexer nodes. The daily results are showing 99% where as actual disk usage is much lower. Can you help clarify. I will submit the actual support contract later.
Thanks,
Naren
| rest splunk_server_group=dmc_group_* /services/server/status/partitions-space
| eval free = if(isnotnull(available), available, free)
| eval usage = capacity - free
| eval pct_usage = floor(usage / capacity * 100)
| where pct_usage > 92
| stats first(fs_type) as fs_type first(capacity) AS capacity first(usage) AS usage first(pct_usage) AS pct_usage by splunk_server, mount_point
| eval usage = round(usage / 1024, 2)
| eval capacity = round(capacity / 1024, 2)
| rename splunk_server AS Instance mount_point as "Mount Point", fs_type as "File System Type", usage as "Usage (GB)", capacity as "Capacity (GB)", pct_usage as "Usage (%)"
Alert search Results:
Instance Mount Point File System Type Capacity (GB) Usage (GB) Usage (%)
prd-sjc-splunk-indexer-1 /opt/colddb ext4 14881.80 14239.33 95
prd-sjc-splunk-indexer-2 /opt/colddb ext4 14881.80 14523.47 97
prd-sjc-splunk-indexer-3 /opt/colddb ext4 14881.80 14664.28 98
prd-sjc-splunk-indexer-4 /opt/colddb ext4 14881.80 14845.24 99
prd-sjc-splunk-indexer-5 /opt/colddb ext4 14881.80 14612.96 98
prd-sjc-splunk-indexer-6 /opt/colddb ext4 14881.80 14744.09 99
Actual Disk space:
Processing on prd-sjc-splunk-indexer-2
/dev/mapper/hot-hot 10403135808 7814205760 2064642256 80% /opt/splunk
/dev/mapper/cold-cold 15604702004 10252296568 4565973644 70% /opt/colddb
Processing on prd-sjc-splunk-indexer-1:
/dev/mapper/hot-hot 10403135808 7956960136 1921887880 81% /opt/splunk
/dev/mapper/cold-cold 15604702004 9749420004 5068850208 66% /opt/colddb
Processing on prd-sjc-splunk-indexer-5
/dev/xvdg 10403139904 7912240516 1966611388 81% /opt/splunk
/dev/mapper/colddb-colddb 15604697908 9752163196 5066103124 66% /opt/colddb
Processing on prd-sjc-splunk-indexer-3:
/dev/mapper/hot-hot 10403135808 7865249624 2013598392 80% /opt/splunk
/dev/mapper/cold-cold 15604702004 9997688028 4820582184 68% /opt/colddb
Processing on prd-sjc-splunk-indexer-4
/dev/mapper/cold-colddb 15604697908 10681093532 4137236996 73% /opt/colddb
/dev/mapper/hot-hotdb 10403135808 7779513904 2099334112 79% /opt/splunk
Processing on prd-sjc-splunk-indexer-6
/dev/xvdg 10321219904 7766226312 2030705592 80% /opt/splunk
/dev/mapper/colddb-colddb 15604697908 10138947556 4679318764 69% /opt/colddb
Processing on prd-sjc-splunk-indexer-7
/dev/xvdg 10403139904 7783278992 2095572912 79% /opt/splunk
/dev/xvdh 17111506844 1754748380 14497765008 11% /opt/colddb
updated to mark code and data as code
... View more