Monitoring Splunk

How do I find the disk utilization on all my indexes

SamHTexas
Contributor

How do I find the disk utilization on all my indexes. How do I write an alert for each going over a certain amount?

Labels (1)
Tags (1)
0 Karma
1 Solution

soutamo
SplunkTrust
SplunkTrust

Hi

We are using this (based on some MC queries) to see index space usage (we have also separate query for volumes).

| rest splunk_server_group=dmc_group_indexer splunk_server_group=* /services/data/indexes splunk_server="<YOUR INDEXERS>" 
| join type=outer splunk_server title 
    [| rest splunk_server_group=dmc_group_indexer splunk_server_group=* /services/data/indexes-extended splunk_server="<YOUR INDEXERS>"] 
| eval _dmc_title=if(isnotnull(title),title,'data.name') 
| where (true() XOR ((('_dmc_title' == "history") OR ('_dmc_title' == "_thefishbucket")) OR ('_dmc_title' == "_blocksignature"))) 
| fields - _dmc_title 
| eval elapsedTime=(now() - strptime(minTime,"%Y-%m-%dT%H:%M:%S%z")) 
| eval dataAge=ceil((elapsedTime / 86400)) 
| eval indexSizeGB=if((currentDBSizeMB > 1),(currentDBSizeMB / 1024),null()) 
| eval maxSizeGB=(maxTotalDataSizeMB / 1024) 
| eval sizeUsagePerc=((indexSizeGB / maxSizeGB) * 100) 
| stats dc(splunk_server) AS Instances count(indexSizeGB) as "Non-Empty Instances" sum(indexSizeGB) AS totalSize avg(indexSizeGB) as averageSize avg(sizeUsagePerc) as averageSizePerc max(sizeUsagePerc) as maxSizePerc count(eval(sizeUsagePerc > 95)) as instancesFreezingDueToSize median(dataAge) as medianDataAge max(dataAge) as oldestDataAge count(eval(elapsedTime > frozenTimePeriodInSecs)) as instancesFreezingDueToAge sum(frozenTimePeriodInSecs) as infiniteFreezingFlag by title 
| eval totalSize=if(isnotnull(totalSize),round(totalSize,2),0) 
| eval averageSize=if(isnotnull(averageSize),round(averageSize,2),0) 
| eval averageSizePerc=if(isnotnull(averageSizePerc),(round(averageSizePerc,2) . "%"),"N/A") 
| eval maxSizePerc=if(isnotnull(maxSizePerc),(round(maxSizePerc,2) . "%"),"N/A") 
| eval instancesFreezingDueToSize=if((averageSizePerc != "N/A"),instancesFreezingDueToSize,"N/A") 
| eval medianDataAge=if(isnum(medianDataAge),medianDataAge,"N/A") 
| eval oldestDataAge=if(isnum(oldestDataAge),oldestDataAge,"N/A") 
| eval instancesFreezingDueToAge=if((infiniteFreezingFlag > 0),instancesFreezingDueToAge,"N/A") 
| rename title as Index, totalSize as "Total Size (GB)", averageSize as "Average Size (GB)", averageSizePerc as "Average Usage (%)", maxSizePerc as "max Usage (%)", instancesFreezingDueToSize as "Instances Freezing Due To Size*", medianDataAge as "Median Data Age (days)", oldestDataAge as "Oldest Data Age (days)", instancesFreezingDueToAge as "Instances Freezing Due to Age" 
| fields - infiniteFreezingFlag 
| search "Non-Empty Instances">2 
| fields - "Non-Empty Instances" 
| sort - "Average Usage (%)"

This works on MC.

r. Ismo 

View solution in original post

soutamo
SplunkTrust
SplunkTrust

Hi

We are using this (based on some MC queries) to see index space usage (we have also separate query for volumes).

| rest splunk_server_group=dmc_group_indexer splunk_server_group=* /services/data/indexes splunk_server="<YOUR INDEXERS>" 
| join type=outer splunk_server title 
    [| rest splunk_server_group=dmc_group_indexer splunk_server_group=* /services/data/indexes-extended splunk_server="<YOUR INDEXERS>"] 
| eval _dmc_title=if(isnotnull(title),title,'data.name') 
| where (true() XOR ((('_dmc_title' == "history") OR ('_dmc_title' == "_thefishbucket")) OR ('_dmc_title' == "_blocksignature"))) 
| fields - _dmc_title 
| eval elapsedTime=(now() - strptime(minTime,"%Y-%m-%dT%H:%M:%S%z")) 
| eval dataAge=ceil((elapsedTime / 86400)) 
| eval indexSizeGB=if((currentDBSizeMB > 1),(currentDBSizeMB / 1024),null()) 
| eval maxSizeGB=(maxTotalDataSizeMB / 1024) 
| eval sizeUsagePerc=((indexSizeGB / maxSizeGB) * 100) 
| stats dc(splunk_server) AS Instances count(indexSizeGB) as "Non-Empty Instances" sum(indexSizeGB) AS totalSize avg(indexSizeGB) as averageSize avg(sizeUsagePerc) as averageSizePerc max(sizeUsagePerc) as maxSizePerc count(eval(sizeUsagePerc > 95)) as instancesFreezingDueToSize median(dataAge) as medianDataAge max(dataAge) as oldestDataAge count(eval(elapsedTime > frozenTimePeriodInSecs)) as instancesFreezingDueToAge sum(frozenTimePeriodInSecs) as infiniteFreezingFlag by title 
| eval totalSize=if(isnotnull(totalSize),round(totalSize,2),0) 
| eval averageSize=if(isnotnull(averageSize),round(averageSize,2),0) 
| eval averageSizePerc=if(isnotnull(averageSizePerc),(round(averageSizePerc,2) . "%"),"N/A") 
| eval maxSizePerc=if(isnotnull(maxSizePerc),(round(maxSizePerc,2) . "%"),"N/A") 
| eval instancesFreezingDueToSize=if((averageSizePerc != "N/A"),instancesFreezingDueToSize,"N/A") 
| eval medianDataAge=if(isnum(medianDataAge),medianDataAge,"N/A") 
| eval oldestDataAge=if(isnum(oldestDataAge),oldestDataAge,"N/A") 
| eval instancesFreezingDueToAge=if((infiniteFreezingFlag > 0),instancesFreezingDueToAge,"N/A") 
| rename title as Index, totalSize as "Total Size (GB)", averageSize as "Average Size (GB)", averageSizePerc as "Average Usage (%)", maxSizePerc as "max Usage (%)", instancesFreezingDueToSize as "Instances Freezing Due To Size*", medianDataAge as "Median Data Age (days)", oldestDataAge as "Oldest Data Age (days)", instancesFreezingDueToAge as "Instances Freezing Due to Age" 
| fields - infiniteFreezingFlag 
| search "Non-Empty Instances">2 
| fields - "Non-Empty Instances" 
| sort - "Average Usage (%)"

This works on MC.

r. Ismo 

View solution in original post

SamHTexas
Contributor

Thank u very much for your help.

Tags (1)
0 Karma

SamHTexas
Contributor

This does not work on my Cluster master. Did not know how to trouble shoot it. Any advice?

 

Tags (1)
0 Karma

soutamo
SplunkTrust
SplunkTrust
It must be a Monitoring Console not a cluster master (unless you use it as MC too).
0 Karma

SamHTexas
Contributor

There is something am not doing right. I ran it in my search & reporting inside the Monitoring console. Still not producing any results. Please advise when free please.

Tags (1)
0 Karma

soutamo
SplunkTrust
SplunkTrust
Have you change <YOUR INDEXERS> to correct values or use * there?
0 Karma

SamHTexas
Contributor

Would you be kind to share the one for the volumes as well. I'd appreciate if you'd share what SPLs you use daily to get a Splunk / ES heart beat making sure it is alive. Thank u for your time sir.

0 Karma

soutamo
SplunkTrust
SplunkTrust

Query for volumes

| rest services/data/index-volumes splunk_server="*" 
| eval _dmc_volume = if(isnotnull(title), title, 'data.name') 
| where NOT _dmc_volume == "_splunk_summaries" 
| fields - _dmc_volume 
| eval volumeSizeGB = if(total_size > 1, round(total_size / 1024, 2), null()) 
| eval sizeUsagePerc = round(total_size / max_size * 100,1) 
| stats dc(splunk_server) as Instances count(eval(total_size > 1)) as "Non-Empty Instances" sum(volumeSizeGB) as totalSize avg(volumeSizeGB) as avgSize avg(sizeUsagePerc) as avgSizePerc max(sizeUsagePerc) as maxSizePerc count(eval(total_size > max_size)) as volumesFreezingDueToSize by title 
| where 'Non-Empty Instances' > 0 
| fields - "Non-Empty Instances"

 

if needed you should update splunk_server = <YOUR INDEXERS> instead of use * 

0 Karma

SamHTexas
Contributor

Thank u sir for your reply. I will test it out to see what I get. Can this same result be found in monitoring console or via GUI as well?

0 Karma

soutamo
SplunkTrust
SplunkTrust
Not exactly. That was the reason why I did this.
0 Karma

richgalloway
SplunkTrust
SplunkTrust

The dbinspect command will show you how much disk space is used by each bucket.  Add a stats command to show total use by index/indexer.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

SamHTexas
Contributor

I tried 

| dbinspect index=*

or on master

| rest /services/cluster/master/buckets

None worked for me any idea why please?

Tags (1)
0 Karma
Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!