Duplicate Event

[msn2507]

I am extracting logs using REST webservices and its a 3rd party application that maintains the logs. I have to poll every 5 minutes to get the logs. Splunk creates duplicate events for every poll, is there a way to avoid this ?
See the sample log event -
link text

[Ayn]

No, you need to add that logic yourself. For file-based inputs, Splunk keeps track of where to start looking for new events in the file by storing to which position it's already read the file. You need to implement the same kind of thing in your scripted input yourself - find a unique incrementing ID of some kind, then after you've queried the REST webservice store the highest ID and then next time you're querying the webservice compare event ID's against that max ID to make sure you're only getting newer ones.

[msn2507]

I am redirecting the output of the webservice call to file and made it the source for splunk still I am seeing the duplicates. Note: I have checked the option "Follow tail"

Any help is appreciated