Splunk Dev

How do I filter data with Python/REST KV Store query ?

goodsellt
Contributor

I'm attempting to do a query to a KV store with the Python SDK's methods. I'm having an issue with passing along an operator and filter properly. The query I'm sending to the KV store is:

{"$lte":{"Last_Update":str(age)}} 

Where Last_Update is an epoch time string inside of my KV store, and age is var for an epoch time I'm trying to filter against (generated by "age = int(time.time())-86400"

When I run the query, I'm merely returned the entire KV store instead of the filtered data I desire.

I've tried setting age as an int, and trying to format the query differently, however it has not had any effect on the results I'm seeing. If I do a splunk query of "|inputlookup my_kv_store where "age" (actual value from python) <= Last_Update" does work correctly though and gives me the results I'd expect.

Anyone with any experience using this have any ideas where I'm going wrong? Unfortunately there doesn't appear to be any examples in the documentation of using the $lt, $lte, $gt, and $gte functions, nor within the framework of the Python SDK vs a standard REST call.

1 Solution

lweber
Path Finder

try it the other way around 😉

{"Last_Update":{"$lte":str(age)}} 

View solution in original post

0 Karma

lweber
Path Finder

try it the other way around 😉

{"Last_Update":{"$lte":str(age)}} 
0 Karma

goodsellt
Contributor

Thanks! This is correct in addition to another change I made based on the examples for the KVStore query (differs from the documentation on the site):

def cleanup(collection):
    '''This function scrubs the table of records which have not been updated in the last 24 hours'''
    age = int(time.time())-86400
    clean = {"Last_Update":{"$lte":age}}
    clean = json.dumps(clean)
    res = collection.data.query(query=clean)
    print(len(res))
    #collection.data.delete(query=clean)

lweber
Path Finder

for the documentation you may want to have a look at the mongodb documentation directly: https://docs.mongodb.com/v3.2/reference/operator/query/

but... keep in mind, that the kv store endpoints do not support all of them.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Can you share more of your code please?

0 Karma

goodsellt
Contributor

Sure! This is found in this python function I created, which final goal is designed to clean up old records from the KV Store:

def cleanup(collection):
    '''This function scrubs the table of records which have not been updated in the last 24 hours'''
    age = int(time.time())-86400
    clean = {"$lte":{"Last_Update":str(age)}}
    clean = json.dumps(clean)
    res = collection.data.query(**clean)
    print(len(res))
    #collection.data.delete(**clean)

"collection" is a KVstore found by the service.kvstore function, while the "service" is generated by the splunklib.client.connect function.

0 Karma

jkat54
SplunkTrust
SplunkTrust

and can you link to documentation that explains $lte?

I cant find anything about how to query kvstore via sdk

0 Karma

37dmk
Explorer

i strongly suspect mongodb syntax to be the reason for these queries.

mongodb query syntax 

0 Karma

goodsellt
Contributor

I've been using the following documentation to try and create the KV store query doc:

http://docs.splunk.com/Documentation/Splunk/6.5.0/RESTREF/RESTkvstore

and the following to figure out how to use the KVstore functionalities in the Python SDK:

http://docs.splunk.com/DocumentationStatic/PythonSDK/1.6.0/client.html#splunklib.client.KVStoreColle...

0 Karma

jkat54
SplunkTrust
SplunkTrust

Says you do it this way:

{$lte:{Last_Update:str(age)}} #without quotes around $lte and Last_Update, even though that's not good json imho

right here:

http://docs.splunk.com/Documentation/Splunk/6.5.0/RESTREF/RESTkvstore#Queries

Probably want int(age) instead...

0 Karma

goodsellt
Contributor

Yes in python that is listed as invalid syntax, so you're not able to even build it correctly compared to the example. Also based on my reading of the Python input, it appears that it's looking for a query inside of the dictionary object, but it has no examples of how it expects those to be formatted.

I'm gonna go to Splunk support regarding this to see if there are any internal examples they have.

0 Karma

jkat54
SplunkTrust
SplunkTrust

I typically use requests lib for anything on the API. I know the sdk makes it easy but who knows what it's doing... you have to cut through all the code to figure out whats going on, and then you find its forcing lowercase or url encoding the query etc... who knows. Honestly I cant say that about the splunk python sdk because i've never used it. Just my experience with other SDKs...

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...