Building for the Splunk Platform

Accessing KV Store with Python - collection.data.query()

BernardEAI
Communicator
I'm trying to delete specific items from our kv store by using a python custom command. I retrieve the kv store with the following command:
 
collection = self.service.kvstore[collection_name]
 
I then retrieve all the entries in the kv store with:
 
data_list = collection.data.query()
 
This works correctly, however only 50000 of the entries are returned. Is there a parameter I can pass to query() to remove the limit of 50000?
 
Thanks!
Labels (2)
0 Karma
1 Solution

venkatasri
SplunkTrust
SplunkTrust

Hi @BernardEAI 

limits.conf having 50k limit by default for [kvstore],  I guess you shall change that in conf can not be passed  in query as platform itself limiting it.

max_rows_per_query = <unsigned integer>
* The maximum number of rows that will be returned for a single query to
  a collection.
* If the query returns more rows than the specified value, then returned
  result set will contain the number of rows specified in this value.
* Default: 50000

 --

An upvote would  be appreciated and Accept solution if this reply helps!

View solution in original post

BernardEAI
Communicator

Thanks @venkatasri 

On our DEV server this would be easy to solve,  I could change the max_rows_per_query parameter in limits.conf.

On our production environment, we are tenants on a multi-tenant platform, so we do not have access to the configuration files.

The approach I took here is to make use of the skip parameter that is available in the query function. I can then have a loop that runs through the entire kv store by incrementing the skip parameter:

           while end == False:

                data_list = collection.data.query(skip=skip_tracker)

                if len(data_list) != 0:
                
                    for item in data_list:
                        # perform action on entry in kv store (delete, update etc.)

                    skip_tracker += 50000

                else:
                    end = True

 More details on the query function here:

https://github.com/splunk/splunk-sdk-python/blob/13f07cd08f8b2017c1cdafc2fbc75673013dc713/splunklib/... 

venkatasri
SplunkTrust
SplunkTrust

Hi @BernardEAI 

limits.conf having 50k limit by default for [kvstore],  I guess you shall change that in conf can not be passed  in query as platform itself limiting it.

max_rows_per_query = <unsigned integer>
* The maximum number of rows that will be returned for a single query to
  a collection.
* If the query returns more rows than the specified value, then returned
  result set will contain the number of rows specified in this value.
* Default: 50000

 --

An upvote would  be appreciated and Accept solution if this reply helps!

Get Updates on the Splunk Community!

Using Machine Learning for Hunting Security Threats

REGISTER NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more ...

Security Highlights | November 2022 Newsletter

 November 2022 2022 Gartner Magic Quadrant for SIEM: Splunk Named a Leader for the 9th Year in a RowSplunk is ...

Platform Highlights | November 2022 Newsletter

 November 2022 Skill Up on Splunk with our New Builder Tech Talk SeriesCan you build it? Yes you can! *play ...