How do you calculate the growth of each Index on a monthly basis?

[akarivaratharaj]

We have a requirement to show the data growth of each index on a monthly basis. I tried with the below query from _internal index but it is giving the complete throughput of the index. Instead of that, I need how much storage the indexes are using in total and also the growth trend for each month.

---

index="_internal" host=prod-* log_level=INFO group=per_index_thruput

---

Could anyone please help me to achieve this?

[akarivaratharaj]

Hi @all,

Thankyou so much for your quick responses.
I just wanted to know how the indexes in my splunk system are grown in each month of 2018, a kind of trend. So I have checked the license usage log (which contains the size of indexed data in each index) from my splunk master. But I can see only the last 30 days of index log events.

I just want to clarify that, the log data will be available only for the past 30 days? If that is the case, how can I know that how my index is grown in each month. Is there any other way to get the log details?

Could anyone please help me on this query.

[harsmarvania57]

If you would like to calculate growth of Internal Indexes (Like _internal, _audit and _introspection) then you need to check _introspection index (Query I have provided in my answer) because Internal Indexes do not consume license and due to that you will not able to see it in your license usage but it will still occupy disk space on your Indexer servers.

[akarivaratharaj]

Okay thankyou

[dkeck]

The _internal logs are only stored 30 days by default, you have to increase your retention time if you want to keep them longer. So yes thats correct.

[harsmarvania57]

Hi,

You can use _introspection Index to fetch this information

Please try below query, I am taking average of a day in below query but you can change based on your requirement

index=_introspection (host=INDEXER-1 OR host=INDEXER-2) sourcetype=splunk_disk_objects component=Indexes 
| rename data.* AS *
| eval totalindexsize=total_size+datamodel_summary_size
| eval totalindexsize_GB=(totalindexsize/1024)
| fillnull value=0 totalindexsize_GB
| bin span=1d _time 
| stats avg(totalindexsize_GB) AS Total_Index_Size(GB) by host,name,_time
| convert timeformat="%d-%m-%Y" ctime(_time) AS date
| rename name as IndexName
| table date, host, IndexName, Total_Index_Size(GB)

[tomasmoser]

This is probably the best approach. I am exploring it now. However, as other folks noticed _introspection index does not provide data reflecting reality.

Example:

data.datamodel_summary_size says 2x8GB=16GB (roughly) for given index for two indexers

but real disk size of datamodel_summary subdirectory for the same index shows 2x12GB=24GB total for two indexeres. 

I checked and summary replication is not set to true. 

Would you know why?

[deepashri_123]

Hey@akarivaratharaj,

You can try using dbinspect command
Try something like this

| dbinspect index=*
| stats sum(rawSize) AS rawTotal, sum(sizeOnDiskMB) AS diskTotalinMB by index

Alternatively,
You can try using scripted input to monitor disk space.
https://docs.splunk.com/Documentation/Splunk/7.2.3/AdvancedDev/ScriptSetup

Let me know if this helps!!

[akarivaratharaj]

Hi @deepashri_123 ,

I can see the field sizeOnDiskMB have data in MB. So this field denotes the total size used by each indexes.
How about rawSize? Is this also in MB? Is it denoting the total size allocated to each index? Right?

[dkeck]

Hi,

would this be what you were looking for?

index=_internal source=*license_usage.log type=Usage | stats sum(eval(b/1024/1024/1024)) AS volume_b by splunk_server date_mday date_month date_year | stats max(volume_b) by splunk_server date_month date_year

EDIT: sry there was a asterix missing in source. please try again if you allready have

[dkeck]

Please accept the answer if it helped you 🙂 thank you