All Apps and Add-ons

REST API Modular Input: Custom Key-Value Delimiter for Arguments in Custom Python Response Handler

aarontimko
Path Finder

Hi all,

I'm trying to pull in data to Splunk from a REST API which requires t1 and t2 arguments defined in epoch time, milliseconds.
So, if I poll every 5 minutes */5 * * * * for data, I need t2=CurTime and t1=(CurTime - 5min).

That way, I get the past 5 minutes of data every poll.

I set up a custom Response Handler as such, and it seems to be working okay.

class MyCustomAPI:
    '''Response handler for MyCustomAPI.'''

    def __init__(self,**args):
        pass

    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):
        if not "params" in req_args:
            req_args["params"] = {}
        t2 = int(time.time()) * 1000
        t1 = t2 - 300000
        req_args["params"]["t2"] = t2
        req_args["params"]["t1"] = t1
        req_args["params"]["login"] = "myuser"
        req_args["params"]["pass"] = "sadf73jhdsdf34sdf3"
        req_args["params"]["id"] = "754220"
        print_xml_stream(raw_response_output)

Problem:
I actually need to define multiple comma-separated IDs, e.g. 754220,100984,766542. If I enter this into the custom Python class parameter, it appears the URI gets malformed and I get "bad request" from the API I'm hitting. Because the "URL Arguments" in the REST API data input also get populated with comma-separated key-value pairs, I think perhaps that's the problem.

Works:

login=myuser,pass=sadf73jhdsdf34sdf3,id=754220,t1=1440807720000,t2=1440808020000

Probably won't ever work as-is:

login=myuser,pass=sadf73jhdsdf34sdf3,id=754220,100984,766542,t1=1440807720000,t2=1440808020000

I tried putting & as the value for "Delimiter" in the REST API data input, but since I'm not actually defining the values there in the web GUI, but in the Python class, that doesn't seem to help.

Further question for ideal operation:
I would like to only use the MyCustomAPI Python class for defining t1 and t2.
For the rest of the values, I would prefer to enter them on-the-fly in the web GUI.

Response Handler: (only defines t1 and t2)

MyCustomAPI

Response Handler Arguments:

login=myuser&pass=sadf73jhdsdf34sdf3&id=754220,100984,766542

Polling Interval:

*/5 * * * *

Delimiter:

&

Or as such in the attached screenshot... is this possible?

alt text

0 Karma
1 Solution

Damien_Dallimor
Ultra Champion

In rest_ta/bin/rest.py , try changing the dictParameterToStringFormat function to be like :

def dictParameterToStringFormat(parameter):

    if parameter:
        return ''.join(('{}={}'+delimiter).format(key, val) for key, val in parameter.items())[:-1] 
    else:
        return None

View solution in original post

0 Karma

Damien_Dallimor
Ultra Champion

As per previous comment , "Make the delimiter var global then".

alt text

Wait for me to release a new tested version when I get some bandwidth.

0 Karma

aarontimko
Path Finder

Thanks!
I thought my comment didn't go through so that's why it got submitted twice.

0 Karma

Damien_Dallimor
Ultra Champion

In rest_ta/bin/rest.py , try changing the dictParameterToStringFormat function to be like :

def dictParameterToStringFormat(parameter):

    if parameter:
        return ''.join(('{}={}'+delimiter).format(key, val) for key, val in parameter.items())[:-1] 
    else:
        return None
0 Karma

aarontimko
Path Finder

Thanks Damien!

For testing, I temporarily disabled my other REST inputs and changed the dictParameterToStringFormat function to the new code above.

The REST input I left enabled had the comma-separated values in the MyCustomAPI Response Handler and & as the delimiter in the REST API input.

Error below from splunkd.log, I'm running 1.3.9 rest_ta version.
def dictParameterToStringFormat(parameter):

 if parameter:
      return ''.join(('{}={}'+delimiter).format(key, val) for key, val in parameter.items())[:-1]
 else:
      return None

ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" NameError: global name 'delimiter' is not defined
0 Karma

aarontimko
Path Finder

This worked:

  • That change to dictParameterToStringFormat
  • Specify global delimiter
  • Set the delimiter to: &

Now, & becomes the delimiter and the URL arguments look like this:

login=myuser&pass=sadf73jhdsdf34sdf3&id=754220,100984,766542&t1=1440807720000&t2=1440808020000

0 Karma

Damien_Dallimor
Ultra Champion

I'm just giving quick reference examples here.Untested.
Make the delimiter var global then.

0 Karma

Damien_Dallimor
Ultra Champion

I may have to fix some code.

When you set the url args in your response handler using the params dict , they get auto persisted back to inputs.conf for you , and it seems this part of the code is hard coded to use a "," delimiter vs the custom delimiter you have declared to use (&). Give me a day or 2.

alt text

0 Karma
Get Updates on the Splunk Community!

Splunk Cloud | Empowering Splunk Administrators with Admin Config Service (ACS)

Greetings, Splunk Cloud Admins and Splunk enthusiasts! The Admin Configuration Service (ACS) team is excited ...

Tech Talk | One Log to Rule Them All

One log to rule them all: how you can centralize your troubleshooting with Splunk logs We know how important ...

Splunk Security Content for Threat Detection & Response, Q1 Roundup

Join Principal Threat Researcher, Michael Haag, as he walks through: An introduction to the Splunk Threat ...