How do I find the disk utilization on all my indexes

[SamHTexas]

How do I find the disk utilization on all my indexes. How do I write an alert for each going over a certain amount?

[isoutamo · Accepted solution]

Hi

We are using this (based on some MC queries) to see index space usage (we have also separate query for volumes).

| rest splunk_server_group=dmc_group_indexer splunk_server_group=* /services/data/indexes splunk_server="<YOUR INDEXERS>" 
| join type=outer splunk_server title 
    [| rest splunk_server_group=dmc_group_indexer splunk_server_group=* /services/data/indexes-extended splunk_server="<YOUR INDEXERS>"] 
| eval _dmc_title=if(isnotnull(title),title,'data.name') 
| where (true() XOR ((('_dmc_title' == "history") OR ('_dmc_title' == "_thefishbucket")) OR ('_dmc_title' == "_blocksignature"))) 
| fields - _dmc_title 
| eval elapsedTime=(now() - strptime(minTime,"%Y-%m-%dT%H:%M:%S%z")) 
| eval dataAge=ceil((elapsedTime / 86400)) 
| eval indexSizeGB=if((currentDBSizeMB > 1),(currentDBSizeMB / 1024),null()) 
| eval maxSizeGB=(maxTotalDataSizeMB / 1024) 
| eval sizeUsagePerc=((indexSizeGB / maxSizeGB) * 100) 
| stats dc(splunk_server) AS Instances count(indexSizeGB) as "Non-Empty Instances" sum(indexSizeGB) AS totalSize avg(indexSizeGB) as averageSize avg(sizeUsagePerc) as averageSizePerc max(sizeUsagePerc) as maxSizePerc count(eval(sizeUsagePerc > 95)) as instancesFreezingDueToSize median(dataAge) as medianDataAge max(dataAge) as oldestDataAge count(eval(elapsedTime > frozenTimePeriodInSecs)) as instancesFreezingDueToAge sum(frozenTimePeriodInSecs) as infiniteFreezingFlag by title 
| eval totalSize=if(isnotnull(totalSize),round(totalSize,2),0) 
| eval averageSize=if(isnotnull(averageSize),round(averageSize,2),0) 
| eval averageSizePerc=if(isnotnull(averageSizePerc),(round(averageSizePerc,2) . "%"),"N/A") 
| eval maxSizePerc=if(isnotnull(maxSizePerc),(round(maxSizePerc,2) . "%"),"N/A") 
| eval instancesFreezingDueToSize=if((averageSizePerc != "N/A"),instancesFreezingDueToSize,"N/A") 
| eval medianDataAge=if(isnum(medianDataAge),medianDataAge,"N/A") 
| eval oldestDataAge=if(isnum(oldestDataAge),oldestDataAge,"N/A") 
| eval instancesFreezingDueToAge=if((infiniteFreezingFlag > 0),instancesFreezingDueToAge,"N/A") 
| rename title as Index, totalSize as "Total Size (GB)", averageSize as "Average Size (GB)", averageSizePerc as "Average Usage (%)", maxSizePerc as "max Usage (%)", instancesFreezingDueToSize as "Instances Freezing Due To Size*", medianDataAge as "Median Data Age (days)", oldestDataAge as "Oldest Data Age (days)", instancesFreezingDueToAge as "Instances Freezing Due to Age" 
| fields - infiniteFreezingFlag 
| search "Non-Empty Instances">2 
| fields - "Non-Empty Instances" 
| sort - "Average Usage (%)"

This works on MC.

r. Ismo 

[SamHTexas]

Thank u very much for your help.

[SamHTexas]

This does not work on my Cluster master. Did not know how to trouble shoot it. Any advice?

[isoutamo]

It must be a Monitoring Console not a cluster master (unless you use it as MC too).

[SamHTexas]

There is something am not doing right. I ran it in my search & reporting inside the Monitoring console. Still not producing any results. Please advise when free please.

[isoutamo]

Have you change <YOUR INDEXERS> to correct values or use * there?

[SamHTexas]

Would you be kind to share the one for the volumes as well. I'd appreciate if you'd share what SPLs you use daily to get a Splunk / ES heart beat making sure it is alive. Thank u for your time sir.

[isoutamo]

Query for volumes

| rest services/data/index-volumes splunk_server="*" 
| eval _dmc_volume = if(isnotnull(title), title, 'data.name') 
| where NOT _dmc_volume == "_splunk_summaries" 
| fields - _dmc_volume 
| eval volumeSizeGB = if(total_size > 1, round(total_size / 1024, 2), null()) 
| eval sizeUsagePerc = round(total_size / max_size * 100,1) 
| stats dc(splunk_server) as Instances count(eval(total_size > 1)) as "Non-Empty Instances" sum(volumeSizeGB) as totalSize avg(volumeSizeGB) as avgSize avg(sizeUsagePerc) as avgSizePerc max(sizeUsagePerc) as maxSizePerc count(eval(total_size > max_size)) as volumesFreezingDueToSize by title 
| where 'Non-Empty Instances' > 0 
| fields - "Non-Empty Instances"

if needed you should update splunk_server = <YOUR INDEXERS> instead of use * 

[SamHTexas]

Thank u sir for your reply. I will test it out to see what I get. Can this same result be found in monitoring console or via GUI as well?

[isoutamo]

Not exactly. That was the reason why I did this.

[richgalloway]

The dbinspect command will show you how much disk space is used by each bucket.  Add a stats command to show total use by index/indexer.

[SamHTexas]

I tried 

| dbinspect index=*

or on master

| rest /services/cluster/master/buckets

None worked for me any idea why please?