How do I filter data with Python/REST KV Store query ?

[goodsellt]

I'm attempting to do a query to a KV store with the Python SDK's methods. I'm having an issue with passing along an operator and filter properly. The query I'm sending to the KV store is:

{"$lte":{"Last_Update":str(age)}} 

Where Last_Update is an epoch time string inside of my KV store, and age is var for an epoch time I'm trying to filter against (generated by "age = int(time.time())-86400"

When I run the query, I'm merely returned the entire KV store instead of the filtered data I desire.

I've tried setting age as an int, and trying to format the query differently, however it has not had any effect on the results I'm seeing. If I do a splunk query of "|inputlookup my_kv_store where "age" (actual value from python) <= Last_Update" does work correctly though and gives me the results I'd expect.

Anyone with any experience using this have any ideas where I'm going wrong? Unfortunately there doesn't appear to be any examples in the documentation of using the $lt, $lte, $gt, and $gte functions, nor within the framework of the Python SDK vs a standard REST call.

[lweber · Accepted solution]

try it the other way around 😉

{"Last_Update":{"$lte":str(age)}} 

[goodsellt]

Thanks! This is correct in addition to another change I made based on the examples for the KVStore query (differs from the documentation on the site):

def cleanup(collection):
    '''This function scrubs the table of records which have not been updated in the last 24 hours'''
    age = int(time.time())-86400
    clean = {"Last_Update":{"$lte":age}}
    clean = json.dumps(clean)
    res = collection.data.query(query=clean)
    print(len(res))
    #collection.data.delete(query=clean)

[lweber]

for the documentation you may want to have a look at the mongodb documentation directly: https://docs.mongodb.com/v3.2/reference/operator/query/

but... keep in mind, that the kv store endpoints do not support all of them.

[jkat54]

Can you share more of your code please?

[goodsellt]

Sure! This is found in this python function I created, which final goal is designed to clean up old records from the KV Store:

def cleanup(collection):
    '''This function scrubs the table of records which have not been updated in the last 24 hours'''
    age = int(time.time())-86400
    clean = {"$lte":{"Last_Update":str(age)}}
    clean = json.dumps(clean)
    res = collection.data.query(**clean)
    print(len(res))
    #collection.data.delete(**clean)

"collection" is a KVstore found by the service.kvstore function, while the "service" is generated by the splunklib.client.connect function.

[jkat54]

and can you link to documentation that explains $lte?

I cant find anything about how to query kvstore via sdk

[37dmk]

i strongly suspect mongodb syntax to be the reason for these queries.

mongodb query syntax 

[goodsellt]

I've been using the following documentation to try and create the KV store query doc:

http://docs.splunk.com/Documentation/Splunk/6.5.0/RESTREF/RESTkvstore

and the following to figure out how to use the KVstore functionalities in the Python SDK:

http://docs.splunk.com/DocumentationStatic/PythonSDK/1.6.0/client.html#splunklib.client.KVStoreColle...

[jkat54]

Says you do it this way:

{$lte:{Last_Update:str(age)}} #without quotes around $lte and Last_Update, even though that's not good json imho

right here:

http://docs.splunk.com/Documentation/Splunk/6.5.0/RESTREF/RESTkvstore#Queries

Probably want int(age) instead...

[goodsellt]

Yes in python that is listed as invalid syntax, so you're not able to even build it correctly compared to the example. Also based on my reading of the Python input, it appears that it's looking for a query inside of the dictionary object, but it has no examples of how it expects those to be formatted.

I'm gonna go to Splunk support regarding this to see if there are any internal examples they have.

[jkat54]

I typically use requests lib for anything on the API. I know the sdk makes it easy but who knows what it's doing... you have to cut through all the code to figure out whats going on, and then you find its forcing lowercase or url encoding the query etc... who knows. Honestly I cant say that about the splunk python sdk because i've never used it. Just my experience with other SDKs...