topic Re: Distributed Architecture: If data is sent to two heavy forwarders, how do you prevent duplicate logs? in Getting Data In

Distributed Architecture: If data is sent to two heavy forwarders, how do you prevent duplicate logs?

Alteek — Mon, 18 Aug 2014 10:11:40 GMT

Hi,

We are moving to a distributed architecture with 1 search head, 1 indexer and 2 heavy forwarders.

The idea is to forward logs from targets (syslog, universal forwarder) to both Heavy Forwarders.
So we ensure that logs are never lost.

But how can we avoid log duplications in the indexer in this case ? Is it handled automatically ?
Or perhaps there is a better way to do it.

Many thanks,
Regards

Re: Distributed Architecture: If data is sent to two heavy forwarders, how do you prevent duplicate logs?

MuS — Tue, 19 Aug 2014 11:32:18 GMT

Hi Alteek,

you can use the load balancing feature of the universal forwarder, check out the docs about Configure forwarders with outputs.conf this way you can avoid event duplication.
Regarding the syslog devices; use a DNS alias or DNS round robin which referees to both heavy forwarders and use this DNS entry as syslog target.

hope this helps to get you started ...

cheers, MuS

Re: Distributed Architecture: If data is sent to two heavy forwarders, how do you prevent duplicate logs?

Alteek — Tue, 19 Aug 2014 15:26:34 GMT

Thank you for your answer.

I'll try to use the load balancing feature of the universal fwd, and I have found some intersting topics about linux heartbeat for the syslog case.

Regards