topic Re: REST API Modular Input: Is there a way to have a REST API call retrieve only new events, not all data? in All Apps and Add-ons

REST API Modular Input: Is there a way to have a REST API call retrieve only new events, not all data?

theouhuios — Tue, 25 Nov 2014 16:18:40 GMT

Hello

I am new to writing Rest APi calls so please pardon my ignorance if its really simple. I wrote a rest api call which talks to cloudera and gets the events. This part works fine. The issue is it polls every 60sec( default) as I am using the Rest API Modular Input and it pulls all data instead of getting just the new events which happened.

Is there a way to tell a REST api call to get only new events? I know that I can have the time in the URL argument which will then keep data limited to only that time, but how to I dynamically change that date time value so that it automatically pulls newer data?

Re: REST API Modular Input: Is there a way to have a REST API call retrieve only new events, not all data?

martin_mueller — Tue, 25 Nov 2014 18:12:40 GMT

If you're on version 1.3.5 then the date token used in the URL should update itself for every call. If you need more tokens you can add them yourself.

Re: REST API Modular Input: Is there a way to have a REST API call retrieve only new events, not all data?

theouhuios — Tue, 25 Nov 2014 20:06:31 GMT

But it only gets the date but not the time right? Each time it runs its getting everything for today instead of getting only the changes which happened.

Re: REST API Modular Input: Is there a way to have a REST API call retrieve only new events, not all data?

martin_mueller — Tue, 25 Nov 2014 20:13:22 GMT

The example token only gives you the date, but you can add any tokens to the app you need.

Re: REST API Modular Input: Is there a way to have a REST API call retrieve only new events, not all data?

theouhuios — Wed, 03 Dec 2014 16:27:05 GMT

When I place a datetime as a token it gets empty events which is actually right. But is it possible to not index the json data if there is no value in the items ?

Re: REST API Modular Input: Is there a way to have a REST API call retrieve only new events, not all data?

martin_mueller — Wed, 03 Dec 2014 19:29:47 GMT

You could set up a transforms.conf rule that routes empty events to the nullQueue.

Re: REST API Modular Input: Is there a way to have a REST API call retrieve only new events, not all data?

theouhuios — Thu, 04 Dec 2014 13:02:18 GMT

Yeah, did the same. I was thinking if I could do it in the input itself, but since that didn't work used the transforms

Re: REST API Modular Input: Is there a way to have a REST API call retrieve only new events, not all data?

martin_mueller — Thu, 04 Dec 2014 18:01:31 GMT

Does that answer your original question?

Re: REST API Modular Input: Is there a way to have a REST API call retrieve only new events, not all data?

theouhuios — Thu, 04 Dec 2014 20:07:29 GMT

Yup. Thanks Martin