REST API Modular Input App - Tokens not resetting

[jatin_patel]

I am using latest version of REST API Modular Input and doing a simple REST data input.

i have added a new tokens in tokens.py shown below which give me two time values, one is the current time value and another one is current - 30 mins values.

def starttime_thirtym():
thirtym = timedelta(minutes=30)
now = datetime.now()
newtime = now - thirtym
return newtime.strftime('%Y-%m-%dT%H:%M:%S')

def timenow():
now = datetime.now()
return now.strftime('%Y-%m-%dT%H:%M:%S')

I then pass these tokens $starttime_thirtym$ and $timenow$ to REST endpoint which will then get me some integer value. I am running this every 30mins. My problem is when it runs first time i am getting value as expected, lets say i run it at 4PM, its passing to values to the REST endpoint 4PM and 3:30PM and lets say i get back value 250. When it runs next time around 4:30, i am still getting the same 250 value. So with every such run i am keep getting the same value. it seems these tokens are not getting rested with every run. I am not sure why. any idea?

Thanks.

[ext_thales_dami]

Hi,

I encountered exactly the same issue and found another solution.

Indeed, using custom response handler worked well to set URL arguments but I didn't find how to use it to set tokens in URL path (ex : http://myendpoint/api/$token$/search).

After analyzing rest_ta/bin/rest.py, I found this part of the script, which is executed for each polling (begins at line 465 in version 1.4 of the REST Modular Input App) :

while True:

    if polling_type == 'cron':
        next_cron_firing = cron_iter.get_next(datetime)
        while get_current_datetime_for_cron() != next_cron_firing:
            time.sleep(float(10))

    for endpoint in endpoint_list:

        if "params" in req_args:
            req_args_params_current = dictParameterToStringFormat(req_args["params"])
        else:
       (...)

The problem is that the endpoint_list variable is set the first time the script runs, but is never updated after that.

I added the functions used when the script starts, before the " for endpoint in endpoint_list " loop as below :

    while True:

        if polling_type == 'cron':
            next_cron_firing = cron_iter.get_next(datetime)
            while get_current_datetime_for_cron() != next_cron_firing:
                time.sleep(float(10))

        original_endpoint=config.get("endpoint")
        #token replacement
        endpoint_list = replaceTokens(original_endpoint)

        for endpoint in endpoint_list:

            if "params" in req_args:
                req_args_params_current = dictParameterToStringFormat(req_args["params"])
            else:
            (...)

After that, tokens are always updated with tokens.py file before the REST API is polled.

Damien, if the answer is correct, is it possible to update the rest.py script for the next version of the REST Modular Input App ?

Thank you in advance, and sorry for my english, I didn't speak for a while 🙂

[quihong]

I like this solution better as it's the expected behavior versus using the Response Handler to set the parameter for the next request (which is confusing).

Wondering what Damien, the author, thinks regarding this solution.

[joaopadilha]

Hi!

We are facing the same issue here.

We are trying to use a token to replace a value in the querystring parameter.
We've written a function on the tokens.py and we've setted the token name in the Endpoint URL: http://www.something.com/getinfo?value=$tokenName$

This didn't work, so we have found this thread and tried the solution you suggested (changing the rest.py to force the token replacement) but this didn't work also...

Thanks

[ext_thales_dami]

Hi,

If your token is not replaced with any value in the URL :

1. What kind of function did you write in tokens.py ?
2. Did you test it (for example by running tokens.py with a print instead of a return in the function) to check that the results are correct ? It should return only a single value (ex : 'zoo') or a list of values (ex : ['goo','foo','zoo'])
3. Does your function have exactly the same name than your token ( def tokenName() ) ?

If your token is replaced with a value the first time but is never updated :

1. Did you restart splunk service after modifying the rest.py script ?
2. Are you sure that your function in tokens.py returns different values each time it is called ?
3. Which version are you using for the REST Modular Input App ? I tried this solution with success only on 1.4 and didn't test it on previous versions.

[joaopadilha]

My token in not replaced with any value in the URL.

1. It's a function that reads and returns a single value from a .csv file
2. Yes we tested it and it works fine.
3. Yes, it has the exact same name.

[ext_thales_dami]

The only case where I didn't have any value in my URL during my tests was because of the variable's format returned by the function.
Did you try something like return str(myvalue) at the end of your function ?

[joaopadilha]

Well, the output from the function is:
'86626496'

[joaopadilha]

Btw, we didn't change the response handler...

[rakeshbhl]

Hi Damien,
I am having similar requirement but in my case I am trying to access a URL to get cookie before endpoint URL. Can you provide some guidelines on how to invoke a pre-check URL / mechanism to check / get cookies before accessing endpoints.
Thanks,
Rakesh

[jatin_patel]

Below is the class i have added in est_ta/bin/responsehandlers.py. Thanks to Damien!!!

class MyClass:

def __init__(self,**args):
    pass

def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):
    print_xml_stream(raw_response_output)
    now = datetime.now()
    thirtym = timedelta(minutes=30)
    newtime = now + thirtym
    time_fmt = '%Y-%m-%dT%H:%M:%S'

    if not "params" in req_args:
        req_args["params"] = {}
    req_args["params"]["starttime_thirtym"] = now.strftime(time_fmt)
    req_args["params"]["timenow"] = newtime.strftime(time_fmt)

[Damien_Dallimor · Accepted solution]

I have a solution for you.

Token substitution only happens once , before the initial request.

To update the minTime and maxTime URL parameters for subsequent requests , you will need to declare a custom response handler in rest_ta/bin/responsehandlers.py

Something like :

class MyAwesomeResponseHandler:

    def __init__(self,**args):
        pass

    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):       

        print_xml_stream(raw_response_output)        

        if not "params" in req_args:
            req_args["params"] = {}

        #implement the methods referred to here
        req_args["params"]["minTime"] = getMinTime()
        req_args["params"]["maxTime"] = getMaxTime()

Then to apply this just declare it in your stanza setup :

[carbdb]

having token in endpoint, and doing that setting in responsehandler.py leads to doubling param in subsequent call:

url addr: https://my.rest.partner/getsomedata?start=$since10min$

url args: <none>

responsehandler: myresphdl

tokens.py: def since10min()

responsehandler.py: def myresphdl() params[since10min] = "xx"

-->

next call calls https://my.rest.partner/getsomedata?start=10&start=20

leading to error from rest-peer

also, if rest-peer throws error,the responsehandler never gets the chance to set the params in reststate.

i solved this by having no tokens used, rest-peer throws error but i ignore it (tweak rest.py), set the responsehandler params and voila, next time it works

is there a better way to achvieve same?

[jatin_patel]

Damien,

Response handler are only after we get the response from REST endpoint and used more to act on the data before it indexes.
But if i am unable to get the data from right time range how does it matter on using this approach?

Also if its keep using the same token value with every run what is the point of defining dynamic tokens in the tokens.py? I could just hardcode the value in the endpoint.

Thanks,
-Jatin.

[Damien_Dallimor]

Response Handlers can serve 2 main purposes ;

1) pre processing of the response before outputting to Splunk for indexing
2) setting parameters for the next request to be made (url args, headers,cookies etc...)

Define your token (as you are currently doing) for the first intial call.

Then use a custom response handler (which sets the minTime maxtime into the next request as per my example) for subsequent calls.

Try it.

[Jasdeep]

Hi @Damien_Dallimor , @jatin_patel ,

I'm doing just like you suggested, defined my function 'testid()'  in tokens.py and responsehandlers.py 

scenario: created a function to read a value from csv file(which is being updated constantly with the last logged AuditId in splunk) and return the value after incrementing.
Tokens.py --->

import csv
def xyz():
with open('abc.csv', 'r') as f:
csv_f=csv.DictReader(f)
for row in csv_f:
x=int(row['AuditId'])
return(str(x+1))

responsehandlers.py ---->

class Mytestclass:

def __init__(self,**args):
pass

def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):
print_xml_stream(raw_response_output)
with open('abc.csv','r') as f:
csv_f = csv.DictReader(f)
for row in csv_f:
x=int(row['AuditId'])

if not "params" in req_args:
req_args["params"]= {}
req_args["params"]["testid"]=str(x+1)

However, I'm still facing some issues,

1. Every event is being logged twice.

2. Some AuditId's are missing.

3.  empty events are getting created with 0 return records.

[jatin_patel]

Now it makes sense...is there a extended documentation for this app? I feel lot of it is not documented in the App page.

[Damien_Dallimor]

This is free community software created and supported by just me. I update the docs from time to time when I can.

[Damien_Dallimor]

Can you post an example of your inputs.conf stanza and also an example of an actual URL you use for your REST request ?

[jatin_patel]

Hi Damien,

Below example will give me count between 5-5:30PM yesterday.

http://internalendpoint.internaldomain.com/service/Service.svc/GetCountInTimeRange?minTime=2015-09-2...

below are the inputs.conf that i have tried
[rest://Order_Data2]
auth_type = none
endpoint = http://internalendpoint.internaldomain.com/service/Service.svc/GetCountInTimeRange?minTime=$starttim...
http_method = GET
index = main
index_error_response_codes = 0
polling_interval = 1800
response_type = text
sequential_mode = 1
sourcetype = TestTotalCount
streaming_request = 0
disabled = 1

[rest://Order_Data_tc2]
auth_type = none
endpoint = http://internalendpoint.internaldomain.com/service/Service.svc/GetCountInTimeRange?minTime=$starttim...
http_method = GET
index = main
index_error_response_codes = 0
polling_interval = */5 * * * *
response_handler = TotalCount
response_type = text
sequential_mode = 0
sourcetype = TestTotalCount2
streaming_request = 0
disabled = 1
sequential_stagger_time =