Splunk Add-on Builder - REST API input pagination

[marco1987]

Hello splunkers,

I'm working with the latest version of Splunk Add-on Builder to index data from a REST API.

TA only pulls the first page of results by calling:

https://mywebpage.com/api/source/v2

At the bottom of the pulled data are URL for the next url:

"next_url" : "/api/source/v2?last=5431"

How do I configure TA for iterates through all the pages?

I checked from link below, but i dont' understand how (or if is possible) pass the variable  from modular input to my endpoint like this or in other way:

https://mywebpage.com/api/source/v2?last=${next_url}

https://docs.splunk.com/Documentation/AddonBuilder/4.3.0/UserGuide/ConfigureDataCollection#Pass_valu... 

Any ideas?

Thanks!

[marnall]

I recommend coding your modular input. You can use the page number as a checkpoint, then index a page and increment or decrement the checkpoint. Set an interval so that your input gets a page every X seconds, then has a condition to stop when the checkpoint gets to the end.