Archive
Highlighted

How do you calculate the growth of each Index on a monthly basis?

Path Finder

We have a requirement to show the data growth of each index on a monthly basis. I tried with the below query from _internal index but it is giving the complete throughput of the index. Instead of that, I need how much storage the indexes are using in total and also the growth trend for each month.


index="_internal" host=prod-* log_level=INFO group=per_index_thruput

Could anyone please help me to achieve this?

Tags (2)
0 Karma
Highlighted

Re: How do you calculate the growth of each Index on a monthly basis?

Influencer

Hi,

would this be what you were looking for?

index=_internal source=*license_usage.log type=Usage | stats sum(eval(b/1024/1024/1024)) AS volume_b by splunk_server date_mday date_month date_year | stats max(volume_b) by splunk_server date_month date_year

EDIT: sry there was a asterix missing in source. please try again if you allready have

Highlighted

Re: How do you calculate the growth of each Index on a monthly basis?

Influencer

Please accept the answer if it helped you 🙂 thank you

0 Karma
Highlighted

Re: How do you calculate the growth of each Index on a monthly basis?

Motivator

Hey@akarivaratharaj,

You can try using dbinspect command
Try something like this

| dbinspect index=*
| stats sum(rawSize) AS rawTotal, sum(sizeOnDiskMB) AS diskTotalinMB by index

Alternatively,
You can try using scripted input to monitor disk space.
https://docs.splunk.com/Documentation/Splunk/7.2.3/AdvancedDev/ScriptSetup

Let me know if this helps!!

0 Karma
Highlighted

Re: How do you calculate the growth of each Index on a monthly basis?

Path Finder

Hi @deepashri_123 ,

I can see the field sizeOnDiskMB have data in MB. So this field denotes the total size used by each indexes.
How about rawSize? Is this also in MB? Is it denoting the total size allocated to each index? Right?

0 Karma
Highlighted

Re: How do you calculate the growth of each Index on a monthly basis?

SplunkTrust
SplunkTrust

Hi,

You can use _introspection Index to fetch this information

Please try below query, I am taking average of a day in below query but you can change based on your requirement

index=_introspection (host=INDEXER-1 OR host=INDEXER-2) sourcetype=splunk_disk_objects component=Indexes 
| rename data.* AS *
| eval totalindexsize=total_size+datamodel_summary_size
| eval totalindexsize_GB=(totalindexsize/1024)
| fillnull value=0 totalindexsize_GB
| bin span=1d _time 
| stats avg(totalindexsize_GB) AS Total_Index_Size(GB) by host,name,_time
| convert timeformat="%d-%m-%Y" ctime(_time) AS date
| rename name as IndexName
| table date, host, IndexName, Total_Index_Size(GB)
0 Karma
Highlighted

Re: How do you calculate the growth of each Index on a monthly basis?

Path Finder

Hi @all,

Thankyou so much for your quick responses.
I just wanted to know how the indexes in my splunk system are grown in each month of 2018, a kind of trend. So I have checked the license usage log (which contains the size of indexed data in each index) from my splunk master. But I can see only the last 30 days of index log events.

I just want to clarify that, the log data will be available only for the past 30 days? If that is the case, how can I know that how my index is grown in each month. Is there any other way to get the log details?

Could anyone please help me on this query.

0 Karma
Highlighted

Re: How do you calculate the growth of each Index on a monthly basis?

Influencer

The _internal logs are only stored 30 days by default, you have to increase your retention time if you want to keep them longer. So yes thats correct.

0 Karma
Highlighted

Re: How do you calculate the growth of each Index on a monthly basis?

SplunkTrust
SplunkTrust

If you would like to calculate growth of Internal Indexes (Like _internal, _audit and _introspection) then you need to check _introspection index (Query I have provided in my answer) because Internal Indexes do not consume license and due to that you will not able to see it in your license usage but it will still occupy disk space on your Indexer servers.

0 Karma
Highlighted

Re: How do you calculate the growth of each Index on a monthly basis?

Path Finder

Okay thankyou

0 Karma