REST API Modular Input: Is there a way to have a REST API call retrieve only new events, not all data?

[theouhuios]

Hello

I am new to writing Rest APi calls so please pardon my ignorance if its really simple. I wrote a rest api call which talks to cloudera and gets the events. This part works fine. The issue is it polls every 60sec( default) as I am using the Rest API Modular Input and it pulls all data instead of getting just the new events which happened.

Is there a way to tell a REST api call to get only new events? I know that I can have the time in the URL argument which will then keep data limited to only that time, but how to I dynamically change that date time value so that it automatically pulls newer data?

[martin_mueller · Accepted solution]

If you're on version 1.3.5 then the date token used in the URL should update itself for every call. If you need more tokens you can add them yourself.

[martin_mueller]

Does that answer your original question?

[theouhuios]

Yup. Thanks Martin

[martin_mueller]

You could set up a transforms.conf rule that routes empty events to the nullQueue.

[theouhuios]

Yeah, did the same. I was thinking if I could do it in the input itself, but since that didn't work used the transforms

[theouhuios]

When I place a datetime as a token it gets empty events which is actually right. But is it possible to not index the json data if there is no value in the items ?

[martin_mueller]

The example token only gives you the date, but you can add any tokens to the app you need.

[theouhuios]

But it only gets the date but not the time right? Each time it runs its getting everything for today instead of getting only the changes which happened.