topic Re: Duplicate Event in Getting Data In

Duplicate Event

msn2507 — Fri, 31 May 2013 04:14:06 GMT

I am extracting logs using REST webservices and its a 3rd party application that maintains the logs. I have to poll every 5 minutes to get the logs. Splunk creates duplicate events for every poll, is there a way to avoid this ?
See the sample log event -
link text

Re: Duplicate Event

Ayn — Fri, 31 May 2013 07:35:44 GMT

No, you need to add that logic yourself. For file-based inputs, Splunk keeps track of where to start looking for new events in the file by storing to which position it's already read the file. You need to implement the same kind of thing in your scripted input yourself - find a unique incrementing ID of some kind, then after you've queried the REST webservice store the highest ID and then next time you're querying the webservice compare event ID's against that max ID to make sure you're only getting newer ones.

Re: Duplicate Event

msn2507 — Fri, 05 Jul 2013 05:41:14 GMT

I am redirecting the output of the webservice call to file and made it the source for splunk still I am seeing the duplicates. Note: I have checked the option "Follow tail"

Any help is appreciated