Big Indexes best practice

[rayar]

Hi

We have very big indexes (300 GB ) 

Also we have very limited  storage 

is it recommended to split the index to smaller indexes (storage , performance )  ?

[richgalloway]

Using more, smaller indexes will not solve a storage limit problem. In fact, it may make it worse because of the additional metadata needed by Splunk and the OS to store the added indexes.  Be smart about how you index data, however.  Don't put everything into a single index.  Use a new index when access or retention rules demand it.

Some possible solutions:

1. As @to4kawa suggested, use volumes to help prevent the storage system from filling up.
2. Add more storage.
3. Index less data.
4. Retain your data for shorter times.

[isoutamo]

Yes, definitely use volumes! That will save your days many times. 
Try to minimize count of buckets and use auto_high_volumes unless you are using smart store. 
And remember IOPS are your friends. 

[rayar]

Thanks a lot for your inputs 

what about the searches will it improve the search performance to split the data to separate index or the performance will be the same if I filter the huge index with index =www sourcetype=yyy ?

[richgalloway]

You can improve search performance by splitting the data among more indexers (servers).

Using more smaller indexes may or may not help since there are other considerations such as the nature of your data and the nature of your searches.  Having to open and unzip many buckets could slow down searches.  OTOH, finding data in a very large index can also be slow.  There's a trade-off and finding the best balance will take some experimentation.

[rayar · Accepted solution]

Thanks a lot 

We will try to play with it 

[isoutamo]

Hi

it depends what kind of data you have on that index (hosts, source types, sources, cardinality) and how much your daily ingesting volume is? In rule of thumb 300GB didn’t sound much if there is e.g 1 month data and it’s from several sources. I liked to say that probably this amount of data don’t needs any special arrangements yet. When you are talking to 300GB / day / 1 source / host / source type then there maybe a need for some arrangements based on your queries and analyse needs. 

r. Ismo

[to4kawa]

https://www.aditumpartners.com/3-splunk-best-practices-i-learned-the-hard-way/

I'd like to know too.