Why is Syndication input repeating events?

jovelfer — Tue, 01 Mar 2022 17:00:32 GMT

Hi all,

I'm using the syndication component (latest version), to fetch data from multiple feeds:

https://www.cloudflarestatus.com/history.atom
https://cloud.ibm.com/status/api/notifications/feed.rss
https://status.aws.amazon.com/rss/all.rss
https://status.cloud.google.com/feed.atom
https://ocistatus.oraclecloud.com/history.rss

By adding the entries, the events have started to repeat every time each feed is processed, which is 5 minutes, that is, it is re-indexing the entire set of events every 5 minutes for each feed. The check is activated so that it only takes into account new events.

When I set one feed, for example google feed with 3 events:

After 5 min:

If I make:

index=gcc_extension_1 source = syndication://google_gcc_ext | stats count values(host) values(source) values(sourcetype) values(index) by _raw | WHERE count>0

There are 6 results, note that it is not the entire _raw that is repeated, since the _indextime is different each time the array is processed.

I've been researching and doing all kinds of tests for a long time, but I don't know what the problem could be. If anyone could help me out a bit with this I'd really appreciate it.

Here, the detail of feed conf:

Aside from screenshots, I can provide configuration as needed.

Thank you very much in advance.

Re: Syndication input repeating events- How do I fix this?

jovelfer — Tue, 01 Mar 2022 16:16:25 GMT

Any update on this?

topic Re: Syndication input repeating events- How do I fix this? in Getting Data In

Why is Syndication input repeating events?

Re: Syndication input repeating events- How do I fix this?