I built a simple TA using the Add-On Builder (v3.0.1) on Splunk 8.0.4. Once thing I noticed is that the TA only pulls the first page of results and at the bottom of the pulled data are URLs for the existing, next, and last pages.
How do I configure this so that my TA iterates through all the pages? I thought it would be using the checkpoint settings, but I'm not sure how to set that up. I am not a developer so this endeavor has been a tough learning experience. Splunk documentation lacks examples for these things.
The API I am pulling from says I can use page[number] and page[size] but I don't know how to have the TA pull from 1 to last (6 in this case).
Existing Setup
REST URL:
https://api.website.com/${rest_endpoint}?page[number]=${page_number}
Event Extraction settings
JSON path
$.data
Checkpoint settings
Checkpoint parameter name: page_number
Checkpoint field path: $.links.last
Checkpoint initial value: 1
My Data
{
"data":[
{
"id":"52",
"type":"report",
"attributes":{
"name":"Bonorum",
"description":"Lorem ipsum dolor sit amet.",
"created_at":"2019-01-05T01:51:19.000Z"
}
},
{
"id":"7",
"type":"report",
"attributes":{
"name":"Perspiciatis",
"description":"Quia dolor sit amet.",
"created_at":"2017-01-05T01:51:19.000Z"
}
}
],
"links":{
"self":"https://api.website.com/reports?page%5Bnumber%5D=1",
"next":"https://api.website.com/reports?page%5Bnumber%5D=2",
"last":"https://api.website.com/reports?page%5Bnumber%5D=6"
}
}