I'm running into a strange issue with checkpointing--and it seems to have to do with the JSON array returning events in no clear order.
The REST URL I'm querying looks like this:
https://RESTURL.com/api/incidents?updated_after=2021-04-25T12:00:00Z
Sample output:
[
{
"id":847,
"summary":"test",
"updated_at":"2021-04-25T12:23:57Z"
}
{
"id":842,
"summary":"test 2",
"updated_at":"2021-04-26T14:44:55Z"
}
]
If I try to use the "updated_at" time from the last event--using a "Checkpoint field path" like [-1].updated_at--the same event often stays as the last event in the array, even if there are others that are more up-to-date. So the checkpoint doesn't increment. (same issue if I try [0].updated_at )
So with something like the example above, the app will keep querying for updated_after=2021-04-26T14:44:55Z until the order of events happens to randomly change down the line.
Is there a way to use either "JSON path" or "Checkpoint field path" to find the event with the most recent "updated_at" time--and use that as the next checkpoint? Unfortunately there aren't any parameters (per the data source's API documentation) I can use in the REST URL to sort the JSON array.
Any help would be great. Thank you.
