topic Re: REST API Modular Input: how do I make a POST request with variables in the body? in All Apps and Add-ons

REST API Modular Input: how do I make a POST request with variables in the body?

juanmjulio — Mon, 08 Oct 2018 15:41:33 GMT

Hi community,

We have a cloud Service from where we want to pull audit events. This REST API accepts in the request payload the date variable. We don't want to pull events that have already been pulled. This is an example to payload:

{ 
"fromDate": "2018-09-05 00:00:00",     
"toDate": "2018-10-05 23:59:00",     
"product": "XYZ",     
}

We need something like this that changes every day:

fromDate_Variable = today – 2 days
$fromDate_Variable 
2018-09-03 00:00:00

toDate_variable=today – 1 day
$toDate_Variable 
2018-09-04 00:00:00

This payload only pulled events for previous day and are changing every day.

{ 
"fromDate": $ fromDate_Variable,     
"toDate": $toDate_Variable,     
"product": "XYZ",     
}

Could you help me with any suggestions?

Re: REST API Modular Input: how do I make a POST request with variables in the body?

Damien_Dallimor — Mon, 08 Oct 2018 19:44:10 GMT

So you can probably achieve something with a custom response handler.

You will have specified your initial POST payload in your setup config.

Then you can define a custom response handler ie: PostDateHandler , that will update the date values upon each response.

The updated POST payload will also get automatically persisted back to your inputs.conf stanza to survive restarts.

This custom response handler is a class you add to rest_ta/bin/responsehandlers.py

Examples to guide you below.

SEE OTHER ANSWER BELOW.

Re: REST API Modular Input: how do I make a POST request with variables in the body?

durandfr — Fri, 12 Oct 2018 07:27:54 GMT

Thank you Damien - just a shame those examples aren't showing up...I am so keen 😛

Re: REST API Modular Input: how do I make a POST request with variables in the body?

juanmjulio — Tue, 29 Sep 2020 21:36:49 GMT

Hi Damien,

I followed your suggestions, but doesn't worked, I seeing the next error:

10-13-2018 01:21:42.089 -0300 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" post_date["fromDate"] = new_from_date
10-13-2018 01:21:42.089 -0300 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" TypeError: 'str' object does not support item assignment

Could you help me with this?

Re: REST API Modular Input: how do I make a POST request with variables in the body?

Damien_Dallimor — Sat, 13 Oct 2018 06:00:04 GMT

So you can probably achieve something with a custom response handler.

You will have specified your initial POST payload in your setup config.

Then you can define a custom response handler ie: PostDateHandler , that will update the date values upon each response.

The updated POST payload will also get automatically persisted back to your inputs.conf stanza to survive restarts.

This custom response handler is a class you add to rest_ta/bin/responsehandlers.py

Examples to guide you below.

class PostDateHandler:

    def __init__(self,**args):
        pass

    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):       

        #PSEUDO CODE ONLY TO GUIDE YOU , ADJUST AS NECESSARY

        #index HTTP response 
        print_xml_stream(raw_response_output)

        #get POST data
        if not "data" in req_args:
            post_data = {}
        else:
            post_data = json.loads(req_args["data"])

        #set new date to something
        new_from_date = "2018-09-05 00:00:00"
        new_to_date = "2018-10-05 00:00:00"
        post_data["fromDate"] = new_from_date
        post_data["toDate"] = new_to_date

        #update POST data
        req_args["data"] = json.dumps(post_data)

Re: REST API Modular Input: how do I make a POST request with variables in the body?

Damien_Dallimor — Sat, 13 Oct 2018 06:00:36 GMT

Reposted below. For some reason Splunkbase stripped the images and you can't re-add them in edit mode.

Re: REST API Modular Input: how do I make a POST request with variables in the body?

Damien_Dallimor — Sat, 13 Oct 2018 06:07:27 GMT

The example is purely to guide you, it is pseudo code , this is made very clear in the example.

You will have to write your own python. You error indicates your python code is invalid.

We are not going to write the code for you unless you have commercial support.

I updated the pseduo code example below so you can now copy/paste it to get started (it was an image before)

Re: REST API Modular Input: how do I make a POST request with variables in the body?

durandfr — Wed, 17 Oct 2018 16:45:14 GMT

Many thanks Damien for the multiple pointers - now oddly enough, i am witnessing a change in behavior should I be updating the POST data, or not.

If I don't and just parse a json response, defined polling interval is respected - and I can trace a single New scheduled exec process: python /opt/splunk/etc/apps/rest_ta/bin/rest.py once inputs.conf stanza updated from Splunk GUI.
If I do, polling interval is ignored and New scheduled exec process* start cropping up uncontrollably a few times per minute - furthermore, my updated POST json payload gets all messed up.

Re: REST API Modular Input: how do I make a POST request with variables in the body?

juanmjulio — Wed, 17 Oct 2018 19:28:15 GMT

Hi Damien,

My mistake, I had syntax error in my code, now the post are changing but as mentioned durandfr, I seeing same behavior.

Regards

Re: REST API Modular Input: how do I make a POST request with variables in the body?

Damien_Dallimor — Wed, 17 Oct 2018 19:51:59 GMT

Most likely an error in your code / config.

Any logging ?

index=_internal error rest.py

Re: REST API Modular Input: how do I make a POST request with variables in the body?

juanmjulio — Wed, 17 Oct 2018 20:06:25 GMT

I seeing recurrent errors like this, I check conectivity and is ok also I increase the timeout setting but the error persist aleatory way:

ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" Exception performing request: HTTPSConnectionPool(host='xxxxx.fa.us.xxxx.com', port=443): Max retries exceeded with url: /fscmRestApi/fndAuditRESTService/audittrail/getaudithistory?D2=8&D1=7 (Caused by : [Errno 110] Connection timed out)

Regards

Re: REST API Modular Input: how do I make a POST request with variables in the body?

Damien_Dallimor — Wed, 17 Oct 2018 21:10:34 GMT

I would make a calculated guess that you have an error in any custom response handler code you have written.

Re: REST API Modular Input: how do I make a POST request with variables in the body?

juanmjulio — Thu, 18 Oct 2018 01:17:46 GMT

OK, I see that my problem is that I'm using 2 variables as pointer (D1 and D2) to persist this to the inputs.conf even if splunk is restart, I´m using req_args["params"]["D1"] = "something", althoug this are saving in the inputs.conf also append to the end of URL request too, Do you know any way to keep any variable in inputs.conf without append to URL request?

Re: REST API Modular Input: how do I make a POST request with variables in the body?

Damien_Dallimor — Fri, 19 Oct 2018 02:38:12 GMT

The default behavior is to automatically persist any changes to URL Parameters/Cookies/Request Headers/Post Data back to inputs.conf.

And whatever you set any of the above fields to in your custom response handler will be part of the next request in the main code in rest.py

Re: REST API Modular Input: how do I make a POST request with variables in the body?

juanmjulio — Fri, 19 Oct 2018 02:50:53 GMT

Thanks, one more question, How can I update the value of some key in request handler arguments field ?
I know that is possible to get the value as mentioned in https://answers.splunk.com/answers/334524/accessing-response-handler-arguments-from-response-1.html

Re: REST API Modular Input: how do I make a POST request with variables in the body?

Damien_Dallimor — Fri, 19 Oct 2018 03:12:37 GMT

You want to update the value of a key in req_args["params"] ?

if "params" in req_args:
        req_args["params"]["D1"]  = "some new value"

Re: REST API Modular Input: how do I make a POST request with variables in the body?

juanmjulio — Fri, 19 Oct 2018 03:19:51 GMT

Not, I want to update the key=value of response handlers arguments

Re: REST API Modular Input: how do I make a POST request with variables in the body?

Damien_Dallimor — Fri, 19 Oct 2018 03:41:21 GMT

I have no idea what you are asking now.

Post commented code example of what you are trying to accomplish.

Is this what you mean ?

class SomeResponseHandler:

    def __init__(self,**args):
        self.foo = args['foo']
        pass

    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):

        self.foo = "some new value"
        print_xml_stream(raw_response_output)

Re: REST API Modular Input: how do I make a POST request with variables in the body?

juanmjulio — Tue, 29 Sep 2020 21:43:07 GMT

I tested this code but not worked, I want to assign a value to args['foo'] = "some new value" and this value appear in response_handler_args from inputs.conf, and this value could change for this reason I need to modify this with some frecuency.

Re: REST API Modular Input: how do I make a POST request with variables in the body?

Damien_Dallimor — Fri, 19 Oct 2018 05:26:08 GMT

As per previous reply :

The default behavior is to automatically persist any changes to URL Parameters/Cookies/Request Headers/Post Data back to inputs.conf.

Instance variable parameters of response handlers do not get persisted to inputs.conf.
Their intention is for configurable/declarative initialization of response handlers.