topic Re: REST API Modular Input: Custom Key-Value Delimiter for Arguments in Custom Python Response Handler in All Apps and Add-ons

REST API Modular Input: Custom Key-Value Delimiter for Arguments in Custom Python Response Handler

aarontimko — Sat, 29 Aug 2015 01:07:29 GMT

Hi all,

I'm trying to pull in data to Splunk from a REST API which requires t1 and t2 arguments defined in epoch time, milliseconds.
So, if I poll every 5 minutes */5 * * * * for data, I need t2=CurTime and t1=(CurTime - 5min).

That way, I get the past 5 minutes of data every poll.

I set up a custom Response Handler as such, and it seems to be working okay.

class MyCustomAPI:
    '''Response handler for MyCustomAPI.'''

    def __init__(self,**args):
        pass

    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):
        if not "params" in req_args:
            req_args["params"] = {}
        t2 = int(time.time()) * 1000
        t1 = t2 - 300000
        req_args["params"]["t2"] = t2
        req_args["params"]["t1"] = t1
        req_args["params"]["login"] = "myuser"
        req_args["params"]["pass"] = "sadf73jhdsdf34sdf3"
        req_args["params"]["id"] = "754220"
        print_xml_stream(raw_response_output)

Problem:
I actually need to define multiple comma-separated IDs, e.g. 754220,100984,766542. If I enter this into the custom Python class parameter, it appears the URI gets malformed and I get "bad request" from the API I'm hitting. Because the "URL Arguments" in the REST API data input also get populated with comma-separated key-value pairs, I think perhaps that's the problem.

Works:

login=myuser,pass=sadf73jhdsdf34sdf3,id=754220,t1=1440807720000,t2=1440808020000

Probably won't ever work as-is:

login=myuser,pass=sadf73jhdsdf34sdf3,id=754220,100984,766542,t1=1440807720000,t2=1440808020000

I tried putting & as the value for "Delimiter" in the REST API data input, but since I'm not actually defining the values there in the web GUI, but in the Python class, that doesn't seem to help.

Further question for ideal operation:
I would like to only use the MyCustomAPI Python class for defining t1 and t2.
For the rest of the values, I would prefer to enter them on-the-fly in the web GUI.

Response Handler: (only defines t1 and t2)

MyCustomAPI

Response Handler Arguments:

login=myuser&pass=sadf73jhdsdf34sdf3&id=754220,100984,766542

Polling Interval:

*/5 * * * *

Delimiter:

Or as such in the attached screenshot... is this possible?

Re: REST API Modular Input: Custom Key-Value Delimiter for Arguments in Custom Python Response Handler

Damien_Dallimor — Sat, 29 Aug 2015 10:57:07 GMT

I may have to fix some code.

When you set the url args in your response handler using the params dict , they get auto persisted back to inputs.conf for you , and it seems this part of the code is hard coded to use a "," delimiter vs the custom delimiter you have declared to use (&). Give me a day or 2.

Re: REST API Modular Input: Custom Key-Value Delimiter for Arguments in Custom Python Response Handler

Damien_Dallimor — Sun, 30 Aug 2015 10:14:48 GMT

In rest_ta/bin/rest.py , try changing the dictParameterToStringFormat function to be like :

def dictParameterToStringFormat(parameter):

    if parameter:
        return ''.join(('{}={}'+delimiter).format(key, val) for key, val in parameter.items())[:-1] 
    else:
        return None

Re: REST API Modular Input: Custom Key-Value Delimiter for Arguments in Custom Python Response Handler

aarontimko — Mon, 31 Aug 2015 07:27:33 GMT

Thanks Damien!

For testing, I temporarily disabled my other REST inputs and changed the dictParameterToStringFormat function to the new code above.

The REST input I left enabled had the comma-separated values in the MyCustomAPI Response Handler and & as the delimiter in the REST API input.

Error below from splunkd.log, I'm running 1.3.9 rest_ta version.
def dictParameterToStringFormat(parameter):

 if parameter:
      return ''.join(('{}={}'+delimiter).format(key, val) for key, val in parameter.items())[:-1]
 else:
      return None

ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" NameError: global name 'delimiter' is not defined

Re: REST API Modular Input: Custom Key-Value Delimiter for Arguments in Custom Python Response Handler

Damien_Dallimor — Mon, 31 Aug 2015 07:36:37 GMT

I'm just giving quick reference examples here.Untested.
Make the delimiter var global then.

Re: REST API Modular Input: Custom Key-Value Delimiter for Arguments in Custom Python Response Handler

Damien_Dallimor — Mon, 31 Aug 2015 12:49:55 GMT

As per previous comment , "Make the delimiter var global then".

Wait for me to release a new tested version when I get some bandwidth.

Re: REST API Modular Input: Custom Key-Value Delimiter for Arguments in Custom Python Response Handler

aarontimko — Mon, 31 Aug 2015 18:18:36 GMT

Thanks!
I thought my comment didn't go through so that's why it got submitted twice.

Re: REST API Modular Input: Custom Key-Value Delimiter for Arguments in Custom Python Response Handler

aarontimko — Tue, 01 Sep 2015 15:10:14 GMT

This worked:

That change to dictParameterToStringFormat
Specify global delimiter
Set the delimiter to: &

Now, & becomes the delimiter and the URL arguments look like this: