Developing for Splunk Enterprise
Highlighted

How do I filter data with Python/REST KV Store query ?

Contributor

I'm attempting to do a query to a KV store with the Python SDK's methods. I'm having an issue with passing along an operator and filter properly. The query I'm sending to the KV store is:

{"$lte":{"Last_Update":str(age)}} 

Where Last_Update is an epoch time string inside of my KV store, and age is var for an epoch time I'm trying to filter against (generated by "age = int(time.time())-86400"

When I run the query, I'm merely returned the entire KV store instead of the filtered data I desire.

I've tried setting age as an int, and trying to format the query differently, however it has not had any effect on the results I'm seeing. If I do a splunk query of "|inputlookup mykvstore where "age" (actual value from python) <= Last_Update" does work correctly though and gives me the results I'd expect.

Anyone with any experience using this have any ideas where I'm going wrong? Unfortunately there doesn't appear to be any examples in the documentation of using the $lt, $lte, $gt, and $gte functions, nor within the framework of the Python SDK vs a standard REST call.

Highlighted

Re: How do I filter data with Python/REST KV Store query ?

SplunkTrust
SplunkTrust

Can you share more of your code please?

0 Karma
Highlighted

Re: How do I filter data with Python/REST KV Store query ?

Contributor

Sure! This is found in this python function I created, which final goal is designed to clean up old records from the KV Store:

def cleanup(collection):
    '''This function scrubs the table of records which have not been updated in the last 24 hours'''
    age = int(time.time())-86400
    clean = {"$lte":{"Last_Update":str(age)}}
    clean = json.dumps(clean)
    res = collection.data.query(**clean)
    print(len(res))
    #collection.data.delete(**clean)

"collection" is a KVstore found by the service.kvstore function, while the "service" is generated by the splunklib.client.connect function.

0 Karma
Highlighted

Re: How do I filter data with Python/REST KV Store query ?

SplunkTrust
SplunkTrust

and can you link to documentation that explains $lte?

I cant find anything about how to query kvstore via sdk

0 Karma
Highlighted

Re: How do I filter data with Python/REST KV Store query ?

Contributor

I've been using the following documentation to try and create the KV store query doc:

http://docs.splunk.com/Documentation/Splunk/6.5.0/RESTREF/RESTkvstore

and the following to figure out how to use the KVstore functionalities in the Python SDK:

http://docs.splunk.com/DocumentationStatic/PythonSDK/1.6.0/client.html#splunklib.client.KVStoreColle...

0 Karma
Highlighted

Re: How do I filter data with Python/REST KV Store query ?

SplunkTrust
SplunkTrust

Says you do it this way:

{$lte:{LastUpdate:str(age)}} #without quotes around $lte and LastUpdate, even though that's not good json imho

right here:

http://docs.splunk.com/Documentation/Splunk/6.5.0/RESTREF/RESTkvstore#Queries

Probably want int(age) instead...

0 Karma
Highlighted

Re: How do I filter data with Python/REST KV Store query ?

Contributor

Yes in python that is listed as invalid syntax, so you're not able to even build it correctly compared to the example. Also based on my reading of the Python input, it appears that it's looking for a query inside of the dictionary object, but it has no examples of how it expects those to be formatted.

I'm gonna go to Splunk support regarding this to see if there are any internal examples they have.

0 Karma
Highlighted

Re: How do I filter data with Python/REST KV Store query ?

SplunkTrust
SplunkTrust

I typically use requests lib for anything on the API. I know the sdk makes it easy but who knows what it's doing... you have to cut through all the code to figure out whats going on, and then you find its forcing lowercase or url encoding the query etc... who knows. Honestly I cant say that about the splunk python sdk because i've never used it. Just my experience with other SDKs...

0 Karma
Highlighted

Re: How do I filter data with Python/REST KV Store query ?

Path Finder

try it the other way around 😉

{"Last_Update":{"$lte":str(age)}} 

View solution in original post

0 Karma
Highlighted

Re: How do I filter data with Python/REST KV Store query ?

Path Finder

for the documentation you may want to have a look at the mongodb documentation directly: https://docs.mongodb.com/v3.2/reference/operator/query/

but... keep in mind, that the kv store endpoints do not support all of them.

0 Karma