All Apps and Add-ons

Splunk Add-on Builder - REST API input pagination

marco1987
Explorer

Hello splunkers,

I'm working with the latest version of Splunk Add-on Builder to index data from a REST API.

TA only pulls the first page of results by calling:

 

https://mywebpage.com/api/source/v2

 

At the bottom of the pulled data are URL for the next url:

 

"next_url" : "/api/source/v2?last=5431"

 

How do I configure TA for iterates through all the pages?

I checked from link below, but i dont' understand how (or if is possible) pass the variable  from modular input to my endpoint like this or in other way:

 

https://mywebpage.com/api/source/v2?last=${next_url}

 

https://docs.splunk.com/Documentation/AddonBuilder/4.3.0/UserGuide/ConfigureDataCollection#Pass_valu... 

Any ideas?

Thanks!

Labels (2)
0 Karma

marnall
Motivator

I recommend coding your modular input. You can use the page number as a checkpoint, then index a page and increment or decrement the checkpoint. Set an interval so that your input gets a page every X seconds, then has a condition to stop when the checkpoint gets to the end. 

0 Karma
Get Updates on the Splunk Community!

Raise Your Skills at the .conf25 Builder Bar: Your Splunk Developer Destination

Calling all Splunk developers, custom SPL builders, dashboarders, and Splunkbase app creators – the Builder ...

Hunt Smarter, Not Harder: Discover New SPL “Recipes” in Our Threat Hunting Webinar

Are you ready to take your threat hunting skills to the next level? As Splunk community members, you know the ...

Splunk ITSI & Correlated Network Visibility

  Now On Demand   Take Your Network Visibility to the Next Level In today’s complex IT environments, ...