What are KV store size limitations and fix?

[michel_rq]

We are monitoring when a single KV store lookup surpasses 25 GB in size
AND
when the total of all KV store collections surpasses 100 GB in size. 

Time and time again I am seeing collections over 25 GBs and the Total surpasses 100 GBs for many different unique environments. The following doesn't appear to be true.



What are the actual limits for both a single KV Store lookup and the total of all KV Store lookups?
- Can we query them?


We want to prevent any KV store crashes. 

Thank you.

[richgalloway]

What document is shown in the screenshot?  If it's a Splunk document then submit feedback to ask for clarification.

Note the two weasel words in the descriptions: "tested" and "recommended".  These imply the values are not hard limits, but are establishing a point beyond which Splunk says, in effect, "there be dragons".

What collections are getting so large?

[michel_rq]

Thank you for such a quick reply. Check out the second sentence here: 

https://docs.splunk.com/Documentation/ES/7.1.0/Admin/TroubleshootperformancelargeKVStore#Increase_st...

There are various collections. Typically where we keep our list of IOCs.

Im curious if there is a rest endpoint where we can see an actual limit. 

[richgalloway · Accepted solution]

As I suspected, the "Service limits and constraints" document says they are "soft limits".  That means they are not fixed upper bounds, but just the limit which Splunk has tested.  Exceed them at your own risk, but the software will not stop you from doing so.

The lack of documented hard limits and the lack of any limits at all in the REST command responses tell me there is no limit, certainly not a configurable one.