topic Re: Accessing KV Store with Python - collection.data.query() in Splunk Dev

Accessing KV Store with Python - collection.data.query()

BernardEAI — Tue, 27 Jul 2021 08:17:44 GMT

I'm trying to delete specific items from our kv store by using a python custom command. I retrieve the kv store with the following command:

collection = self.service.kvstore[collection_name]

I then retrieve all the entries in the kv store with:

data_list = collection.data.query()

This works correctly, however only 50000 of the entries are returned. Is there a parameter I can pass to query() to remove the limit of 50000?

Thanks!

Re: Accessing KV Store with Python - collection.data.query()

venkatasri — Tue, 27 Jul 2021 09:15:29 GMT

Hi @BernardEAI

limits.conf having 50k limit by default for [kvstore], I guess you shall change that in conf can not be passed in query as platform itself limiting it.

max_rows_per_query = <unsigned integer>
* The maximum number of rows that will be returned for a single query to
  a collection.
* If the query returns more rows than the specified value, then returned
  result set will contain the number of rows specified in this value.
* Default: 50000

An upvote would be appreciated and Accept solution if this reply helps!

Re: Accessing KV Store with Python - collection.data.query()

BernardEAI — Tue, 27 Jul 2021 10:13:52 GMT

Thanks @venkatasri

On our DEV server this would be easy to solve, I could change the max_rows_per_query parameter in limits.conf.

On our production environment, we are tenants on a multi-tenant platform, so we do not have access to the configuration files.

The approach I took here is to make use of the skip parameter that is available in the query function. I can then have a loop that runs through the entire kv store by incrementing the skip parameter:

while end == False: data_list = collection.data.query(skip=skip_tracker) if len(data_list) != 0: for item in data_list: # perform action on entry in kv store (delete, update etc.) skip_tracker += 50000 else: end = True

More details on the query function here:

https://github.com/splunk/splunk-sdk-python/blob/13f07cd08f8b2017c1cdafc2fbc75673013dc713/splunklib/client.py#L3633