All Apps and Add-ons
Highlighted

REST API Modular Input: how do I make a POST request with variables in the body?

New Member

Hi community,

We have a cloud Service from where we want to pull audit events. This REST API accepts in the request payload the date variable. We don't want to pull events that have already been pulled. This is an example to payload:

{ 
"fromDate": "2018-09-05 00:00:00",     
"toDate": "2018-10-05 23:59:00",     
"product": "XYZ",     
}

We need something like this that changes every day:

fromDate_Variable = today – 2 days
$fromDate_Variable 
2018-09-03 00:00:00

toDate_variable=today – 1 day
$toDate_Variable 
2018-09-04 00:00:00

This payload only pulled events for previous day and are changing every day.

{ 
"fromDate": $ fromDate_Variable,     
"toDate": $toDate_Variable,     
"product": "XYZ",     
}

Could you help me with any suggestions?

0 Karma
Highlighted

Re: REST API Modular Input: how do I make a POST request with variables in the body?

Ultra Champion

So you can probably achieve something with a custom response handler.

You will have specified your initial POST payload in your setup config.

Then you can define a custom response handler ie: PostDateHandler , that will update the date values upon each response.

The updated POST payload will also get automatically persisted back to your inputs.conf stanza to survive restarts.

This custom response handler is a class you add to rest_ta/bin/responsehandlers.py

Examples to guide you below.

SEE OTHER ANSWER BELOW.

0 Karma
Highlighted

Re: REST API Modular Input: how do I make a POST request with variables in the body?

New Member

Thank you Damien - just a shame those examples aren't showing up...I am so keen 😛

0 Karma
Highlighted

Re: REST API Modular Input: how do I make a POST request with variables in the body?

Ultra Champion

Reposted below. For some reason Splunkbase stripped the images and you can't re-add them in edit mode.

0 Karma
Highlighted

Re: REST API Modular Input: how do I make a POST request with variables in the body?

New Member

Hi Damien,

I followed your suggestions, but doesn't worked, I seeing the next error:

10-13-2018 01:21:42.089 -0300 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/restta/bin/rest.py" postdate["fromDate"] = newfromdate
10-13-2018 01:21:42.089 -0300 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" TypeError: 'str' object does not support item assignment

Could you help me with this?

0 Karma
Highlighted

Re: REST API Modular Input: how do I make a POST request with variables in the body?

Ultra Champion

The example is purely to guide you, it is pseudo code , this is made very clear in the example.

You will have to write your own python. You error indicates your python code is invalid.

We are not going to write the code for you unless you have commercial support.

I updated the pseduo code example below so you can now copy/paste it to get started (it was an image before)

0 Karma
Highlighted

Re: REST API Modular Input: how do I make a POST request with variables in the body?

New Member

Hi Damien,

My mistake, I had syntax error in my code, now the post are changing but as mentioned durandfr, I seeing same behavior.

Regards

0 Karma
Highlighted

Re: REST API Modular Input: how do I make a POST request with variables in the body?

Ultra Champion

So you can probably achieve something with a custom response handler.

You will have specified your initial POST payload in your setup config.

Then you can define a custom response handler ie: PostDateHandler , that will update the date values upon each response.

The updated POST payload will also get automatically persisted back to your inputs.conf stanza to survive restarts.

This custom response handler is a class you add to rest_ta/bin/responsehandlers.py

Examples to guide you below.

alt text

class PostDateHandler:

    def __init__(self,**args):
        pass

    def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint):       

        #PSEUDO CODE ONLY TO GUIDE YOU , ADJUST AS NECESSARY

        #index HTTP response 
        print_xml_stream(raw_response_output)

        #get POST data
        if not "data" in req_args:
            post_data = {}
        else:
            post_data = json.loads(req_args["data"])

        #set new date to something
        new_from_date = "2018-09-05 00:00:00"
        new_to_date = "2018-10-05 00:00:00"
        post_data["fromDate"] = new_from_date
        post_data["toDate"] = new_to_date

        #update POST data
        req_args["data"] = json.dumps(post_data)
0 Karma
Highlighted

Re: REST API Modular Input: how do I make a POST request with variables in the body?

New Member

Hi Damien,

Happy new year!, Let me tell you that I followed the suggestion and added this in my code, now the update of the data in the request payload field is happening with some frequency. But the frecuency is my problem now, I setted 5 minutes of polling interval but the update of data is happening within the established period more than one time, apparently is updating for each events that arrive. I expecte that the updating happen only when the polling interval is valid.
Any idea how to solve this? Can I use tokens.py to solve this?

Again, thanks for your help!

0 Karma
Highlighted

Re: REST API Modular Input: how do I make a POST request with variables in the body?

New Member

Many thanks Damien for the multiple pointers - now oddly enough, i am witnessing a change in behavior should I be updating the POST data, or not.

  • If I don't and just parse a json response, defined polling interval is respected - and I can trace a single New scheduled exec process: python /opt/splunk/etc/apps/rest_ta/bin/rest.py once inputs.conf stanza updated from Splunk GUI.
  • If I do, polling interval is ignored and New scheduled exec process* start cropping up uncontrollably a few times per minute - furthermore, my updated POST json payload gets all messed up.
0 Karma