Splunk Dev

Accessing KV Store with Python - collection.data.query()

BernardEAI
Communicator
I'm trying to delete specific items from our kv store by using a python custom command. I retrieve the kv store with the following command:
 
collection = self.service.kvstore[collection_name]
 
I then retrieve all the entries in the kv store with:
 
data_list = collection.data.query()
 
This works correctly, however only 50000 of the entries are returned. Is there a parameter I can pass to query() to remove the limit of 50000?
 
Thanks!
Labels (2)
0 Karma
1 Solution

venkatasri
SplunkTrust
SplunkTrust

Hi @BernardEAI 

limits.conf having 50k limit by default for [kvstore],  I guess you shall change that in conf can not be passed  in query as platform itself limiting it.

max_rows_per_query = <unsigned integer>
* The maximum number of rows that will be returned for a single query to
  a collection.
* If the query returns more rows than the specified value, then returned
  result set will contain the number of rows specified in this value.
* Default: 50000

 --

An upvote would  be appreciated and Accept solution if this reply helps!

View solution in original post

BernardEAI
Communicator

Thanks @venkatasri 

On our DEV server this would be easy to solve,  I could change the max_rows_per_query parameter in limits.conf.

On our production environment, we are tenants on a multi-tenant platform, so we do not have access to the configuration files.

The approach I took here is to make use of the skip parameter that is available in the query function. I can then have a loop that runs through the entire kv store by incrementing the skip parameter:

           while end == False:

                data_list = collection.data.query(skip=skip_tracker)

                if len(data_list) != 0:
                
                    for item in data_list:
                        # perform action on entry in kv store (delete, update etc.)

                    skip_tracker += 50000

                else:
                    end = True

 More details on the query function here:

https://github.com/splunk/splunk-sdk-python/blob/13f07cd08f8b2017c1cdafc2fbc75673013dc713/splunklib/... 

venkatasri
SplunkTrust
SplunkTrust

Hi @BernardEAI 

limits.conf having 50k limit by default for [kvstore],  I guess you shall change that in conf can not be passed  in query as platform itself limiting it.

max_rows_per_query = <unsigned integer>
* The maximum number of rows that will be returned for a single query to
  a collection.
* If the query returns more rows than the specified value, then returned
  result set will contain the number of rows specified in this value.
* Default: 50000

 --

An upvote would  be appreciated and Accept solution if this reply helps!

Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...