All Apps and Add-ons

REST API Modular Input: how do I make a POST request with variables in the body?

juanmjulio
New Member

Hi community,

We have a cloud Service from where we want to pull audit events. This REST API accepts in the request payload the date variable. We don't want to pull events that have already been pulled. This is an example to payload:

{ 
"fromDate": "2018-09-05 00:00:00",     
"toDate": "2018-10-05 23:59:00",     
"product": "XYZ",     
}

We need something like this that changes every day:

fromDate_Variable = today – 2 days
$fromDate_Variable 
2018-09-03 00:00:00

toDate_variable=today – 1 day
$toDate_Variable 
2018-09-04 00:00:00

This payload only pulled events for previous day and are changing every day.

{ 
"fromDate": $ fromDate_Variable,     
"toDate": $toDate_Variable,     
"product": "XYZ",     
}

Could you help me with any suggestions?

0 Karma

durandfr
New Member

Many thanks Damien for the multiple pointers - now oddly enough, i am witnessing a change in behavior should I be updating the POST data, or not.

  • If I don't and just parse a json response, defined polling interval is respected - and I can trace a single New scheduled exec process: python /opt/splunk/etc/apps/rest_ta/bin/rest.py once inputs.conf stanza updated from Splunk GUI.
  • If I do, polling interval is ignored and New scheduled exec process* start cropping up uncontrollably a few times per minute - furthermore, my updated POST json payload gets all messed up.
0 Karma

Damien_Dallimor
Ultra Champion

Most likely an error in your code / config.

Any logging ?

index=_internal error rest.py

0 Karma

juanmjulio
New Member

I seeing recurrent errors like this, I check conectivity and is ok also I increase the timeout setting but the error persist aleatory way:

ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" Exception performing request: HTTPSConnectionPool(host='xxxxx.fa.us.xxxx.com', port=443): Max retries exceeded with url: /fscmRestApi/fndAuditRESTService/audittrail/getaudithistory?D2=8&D1=7 (Caused by : [Errno 110] Connection timed out)

Regards

0 Karma

Damien_Dallimor
Ultra Champion

I would make a calculated guess that you have an error in any custom response handler code you have written.

0 Karma

juanmjulio
New Member

OK, I see that my problem is that I'm using 2 variables as pointer (D1 and D2) to persist this to the inputs.conf even if splunk is restart, I´m using req_args["params"]["D1"] = "something", althoug this are saving in the inputs.conf also append to the end of URL request too, Do you know any way to keep any variable in inputs.conf without append to URL request?

0 Karma

Damien_Dallimor
Ultra Champion

The default behavior is to automatically persist any changes to URL Parameters/Cookies/Request Headers/Post Data back to inputs.conf.

And whatever you set any of the above fields to in your custom response handler will be part of the next request in the main code in rest.py

0 Karma

juanmjulio
New Member

Thanks, one more question, How can I update the value of some key in request handler arguments field ?
I know that is possible to get the value as mentioned in https://answers.splunk.com/answers/334524/accessing-response-handler-arguments-from-response-1.html

0 Karma

Damien_Dallimor
Ultra Champion

You want to update the value of a key in req_args["params"] ?

if "params" in req_args:
        req_args["params"]["D1"]  = "some new value"
0 Karma

juanmjulio
New Member

Not, I want to update the key=value of response handlers arguments

0 Karma

Damien_Dallimor
Ultra Champion

I have no idea what you are asking now.

Post commented code example of what you are trying to accomplish.

Is this what you mean ?

class SomeResponseHandler:

    def __init__(self,**args):
        self.foo = args['foo']
        pass

    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):

        self.foo = "some new value"
        print_xml_stream(raw_response_output)
0 Karma

juanmjulio
New Member

I tested this code but not worked, I want to assign a value to args['foo'] = "some new value" and this value appear in response_handler_args from inputs.conf, and this value could change for this reason I need to modify this with some frecuency.

0 Karma

Damien_Dallimor
Ultra Champion

As per previous reply :

The default behavior is to automatically persist any changes to URL Parameters/Cookies/Request Headers/Post Data back to inputs.conf.

Instance variable parameters of response handlers do not get persisted to inputs.conf.
Their intention is for configurable/declarative initialization of response handlers.

0 Karma

Damien_Dallimor
Ultra Champion

So you can probably achieve something with a custom response handler.

You will have specified your initial POST payload in your setup config.

Then you can define a custom response handler ie: PostDateHandler , that will update the date values upon each response.

The updated POST payload will also get automatically persisted back to your inputs.conf stanza to survive restarts.

This custom response handler is a class you add to rest_ta/bin/responsehandlers.py

Examples to guide you below.

alt text

class PostDateHandler:

    def __init__(self,**args):
        pass

    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):       

        #PSEUDO CODE ONLY TO GUIDE YOU , ADJUST AS NECESSARY

        #index HTTP response 
        print_xml_stream(raw_response_output)

        #get POST data
        if not "data" in req_args:
            post_data = {}
        else:
            post_data = json.loads(req_args["data"])

        #set new date to something
        new_from_date = "2018-09-05 00:00:00"
        new_to_date = "2018-10-05 00:00:00"
        post_data["fromDate"] = new_from_date
        post_data["toDate"] = new_to_date

        #update POST data
        req_args["data"] = json.dumps(post_data)
0 Karma

juanmjulio
New Member

Hi Damien,

Happy new year!, Let me tell you that I followed the suggestion and added this in my code, now the update of the data in the request payload field is happening with some frequency. But the frecuency is my problem now, I setted 5 minutes of polling interval but the update of data is happening within the established period more than one time, apparently is updating for each events that arrive. I expecte that the updating happen only when the polling interval is valid.
Any idea how to solve this? Can I use tokens.py to solve this?

Again, thanks for your help!

0 Karma

Damien_Dallimor
Ultra Champion

So you can probably achieve something with a custom response handler.

You will have specified your initial POST payload in your setup config.

Then you can define a custom response handler ie: PostDateHandler , that will update the date values upon each response.

The updated POST payload will also get automatically persisted back to your inputs.conf stanza to survive restarts.

This custom response handler is a class you add to rest_ta/bin/responsehandlers.py

Examples to guide you below.

SEE OTHER ANSWER BELOW.

0 Karma

juanmjulio
New Member

Hi Damien,

I followed your suggestions, but doesn't worked, I seeing the next error:

10-13-2018 01:21:42.089 -0300 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" post_date["fromDate"] = new_from_date
10-13-2018 01:21:42.089 -0300 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" TypeError: 'str' object does not support item assignment

Could you help me with this?

0 Karma

Damien_Dallimor
Ultra Champion

The example is purely to guide you, it is pseudo code , this is made very clear in the example.

You will have to write your own python. You error indicates your python code is invalid.

We are not going to write the code for you unless you have commercial support.

I updated the pseduo code example below so you can now copy/paste it to get started (it was an image before)

0 Karma

juanmjulio
New Member

Hi Damien,

My mistake, I had syntax error in my code, now the post are changing but as mentioned durandfr, I seeing same behavior.

Regards

0 Karma

durandfr
New Member

Thank you Damien - just a shame those examples aren't showing up...I am so keen 😛

0 Karma

Damien_Dallimor
Ultra Champion

Reposted below. For some reason Splunkbase stripped the images and you can't re-add them in edit mode.

0 Karma
Get Updates on the Splunk Community!

Announcing General Availability of Splunk Incident Intelligence!

Digital transformation is real! Across industries, companies big and small are going through rapid digital ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...