topic Splunk Add-on Builder REST data input pagination in Getting Data In

Splunk Add-on Builder REST data input pagination

yotamcp — Wed, 16 Jun 2021 13:10:04 GMT

Hi,

I've recently started working with the Splunk Add-on Builder to quickly query and index data from a REST API of one of my systems.

The response looks something like this:

{ "count":296, "next":"http://mysite.com/api/?page=2&page_size=200", "previous":null, "results": [...] }

I was wondering if there is a way to paginate through the results.

Note: I am aware of checkpoints (I'm using this as well), but am looking for something else.

The behavior I'm looking for is a paginated query, that will continue until "next=null", and then sleep for the specified interval of the data input.

Any ideas? (I would really prefer not implementing a whole new REST client, when there is already a really good implementation here)

Thanks!

Re: Splunk Add-on Builder REST data input pagination

kfennell — Thu, 01 Jul 2021 06:21:56 GMT

Working through the same issue here. I'm wondering if I have to script it myself so it loops through requests until there are no more results. Seems like something that would be common, so I am still thinking there must be configurable way....

Re: Splunk Add-on Builder REST data input pagination

Tyler — Thu, 06 Jan 2022 20:26:04 GMT

The REST data input is quite simple. You'll have to code in order to get it to pull more pages.

This would be a good feature request. I don't see it being difficult to implement.