REST API Modular Input: how do I make a POST request with variables in the body?

[juanmjulio]

Hi community,

We have a cloud Service from where we want to pull audit events. This REST API accepts in the request payload the date variable. We don't want to pull events that have already been pulled. This is an example to payload:

{ 
"fromDate": "2018-09-05 00:00:00",     
"toDate": "2018-10-05 23:59:00",     
"product": "XYZ",     
}

We need something like this that changes every day:

fromDate_Variable = today – 2 days
$fromDate_Variable 
2018-09-03 00:00:00

toDate_variable=today – 1 day
$toDate_Variable 
2018-09-04 00:00:00

This payload only pulled events for previous day and are changing every day.

{ 
"fromDate": $ fromDate_Variable,     
"toDate": $toDate_Variable,     
"product": "XYZ",     
}

Could you help me with any suggestions?

[durandfr]

Many thanks Damien for the multiple pointers - now oddly enough, i am witnessing a change in behavior should I be updating the POST data, or not.

• If I don't and just parse a json response, defined polling interval is respected - and I can trace a single New scheduled exec process: python /opt/splunk/etc/apps/rest_ta/bin/rest.py once inputs.conf stanza updated from Splunk GUI.
• If I do, polling interval is ignored and New scheduled exec process* start cropping up uncontrollably a few times per minute - furthermore, my updated POST json payload gets all messed up.

[Damien_Dallimor]

Most likely an error in your code / config.

Any logging ?

index=_internal error rest.py

[juanmjulio]

I seeing recurrent errors like this, I check conectivity and is ok also I increase the timeout setting but the error persist aleatory way:

ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" Exception performing request: HTTPSConnectionPool(host='xxxxx.fa.us.xxxx.com', port=443): Max retries exceeded with url: /fscmRestApi/fndAuditRESTService/audittrail/getaudithistory?D2=8&D1=7 (Caused by : [Errno 110] Connection timed out)

Regards

[Damien_Dallimor]

I would make a calculated guess that you have an error in any custom response handler code you have written.

[juanmjulio]

OK, I see that my problem is that I'm using 2 variables as pointer (D1 and D2) to persist this to the inputs.conf even if splunk is restart, I´m using req_args["params"]["D1"] = "something", althoug this are saving in the inputs.conf also append to the end of URL request too, Do you know any way to keep any variable in inputs.conf without append to URL request?

[Damien_Dallimor]

The default behavior is to automatically persist any changes to URL Parameters/Cookies/Request Headers/Post Data back to inputs.conf.

And whatever you set any of the above fields to in your custom response handler will be part of the next request in the main code in rest.py

[juanmjulio]

Thanks, one more question, How can I update the value of some key in request handler arguments field ?
I know that is possible to get the value as mentioned in https://answers.splunk.com/answers/334524/accessing-response-handler-arguments-from-response-1.html

[Damien_Dallimor]

You want to update the value of a key in req_args["params"] ?

if "params" in req_args:
        req_args["params"]["D1"]  = "some new value"

[juanmjulio]

Not, I want to update the key=value of response handlers arguments

[Damien_Dallimor]

I have no idea what you are asking now.

Post commented code example of what you are trying to accomplish.

Is this what you mean ?

class SomeResponseHandler:

    def __init__(self,**args):
        self.foo = args['foo']
        pass

    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):

        self.foo = "some new value"
        print_xml_stream(raw_response_output)

[juanmjulio]

I tested this code but not worked, I want to assign a value to args['foo'] = "some new value" and this value appear in response_handler_args from inputs.conf, and this value could change for this reason I need to modify this with some frecuency.

[Damien_Dallimor]

As per previous reply :

The default behavior is to automatically persist any changes to URL Parameters/Cookies/Request Headers/Post Data back to inputs.conf.

Instance variable parameters of response handlers do not get persisted to inputs.conf.
Their intention is for configurable/declarative initialization of response handlers.

[Damien_Dallimor]

So you can probably achieve something with a custom response handler.

You will have specified your initial POST payload in your setup config.

Then you can define a custom response handler ie: PostDateHandler , that will update the date values upon each response.

The updated POST payload will also get automatically persisted back to your inputs.conf stanza to survive restarts.

This custom response handler is a class you add to rest_ta/bin/responsehandlers.py

Examples to guide you below.

class PostDateHandler:

    def __init__(self,**args):
        pass

    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):       

        #PSEUDO CODE ONLY TO GUIDE YOU , ADJUST AS NECESSARY

        #index HTTP response 
        print_xml_stream(raw_response_output)

        #get POST data
        if not "data" in req_args:
            post_data = {}
        else:
            post_data = json.loads(req_args["data"])

        #set new date to something
        new_from_date = "2018-09-05 00:00:00"
        new_to_date = "2018-10-05 00:00:00"
        post_data["fromDate"] = new_from_date
        post_data["toDate"] = new_to_date

        #update POST data
        req_args["data"] = json.dumps(post_data)

[juanmjulio]

Hi Damien,

Happy new year!, Let me tell you that I followed the suggestion and added this in my code, now the update of the data in the request payload field is happening with some frequency. But the frecuency is my problem now, I setted 5 minutes of polling interval but the update of data is happening within the established period more than one time, apparently is updating for each events that arrive. I expecte that the updating happen only when the polling interval is valid.
Any idea how to solve this? Can I use tokens.py to solve this?

Again, thanks for your help!

[Damien_Dallimor]

So you can probably achieve something with a custom response handler.

You will have specified your initial POST payload in your setup config.

Then you can define a custom response handler ie: PostDateHandler , that will update the date values upon each response.

The updated POST payload will also get automatically persisted back to your inputs.conf stanza to survive restarts.

This custom response handler is a class you add to rest_ta/bin/responsehandlers.py

Examples to guide you below.

SEE OTHER ANSWER BELOW.

[juanmjulio]

Hi Damien,

I followed your suggestions, but doesn't worked, I seeing the next error:

10-13-2018 01:21:42.089 -0300 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" post_date["fromDate"] = new_from_date
10-13-2018 01:21:42.089 -0300 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" TypeError: 'str' object does not support item assignment

Could you help me with this?

[Damien_Dallimor]

The example is purely to guide you, it is pseudo code , this is made very clear in the example.

You will have to write your own python. You error indicates your python code is invalid.

We are not going to write the code for you unless you have commercial support.

I updated the pseduo code example below so you can now copy/paste it to get started (it was an image before)

[juanmjulio]

Hi Damien,

My mistake, I had syntax error in my code, now the post are changing but as mentioned durandfr, I seeing same behavior.

Regards

[durandfr]

Thank you Damien - just a shame those examples aren't showing up...I am so keen 😛

[Damien_Dallimor]

Reposted below. For some reason Splunkbase stripped the images and you can't re-add them in edit mode.