I have a concern. I am aware that I can get data from UDP port and send it to an indexer. I have a concern when we have 2 indexers. So how will the setup be like?
1) Sending data to both the indexers
2) Send it to an indexer cluster which will further segregate to indexers.
What I am thinking is If I go with the 1st option, then I will end up with duplicate data as I have replication factor 2.
if you choose the first option you will duplicate your data and you have to manage this situation in your searches (data are duplicated!).
The best way to manage your need is to put a Load Balancer before Indexers to receive UDP data and distribute them to all your Indexers.
In this way you have a fault tolerant architecture, that works also when an Indexer Falls, you don't have duplicate license consumption and you don't have duplicate data.
In addition you can also have clustered Indexers to have available always all your logs.
If you can, it's also better to have two Heavy Forwarders (also with the Load Balancer) to receive UDP logs and forward them to the indexers.
In this way your architecture don't loose logs also with all the Indexers down and during Indexers maintenance.
Well Thanks for the reply Giuseppe
Let me tell you my setup and I am afraid I do not have any scope for any other instance or LB to my setup ,so I have
Total 6 servers
SH =3 (I have LB for these)
Indexers= 2 (in cluster mode)
master/deployment server =1
Do we any other way apart from Indexer LB ?
if you cannot use another LB to distribute UDP logs between the Indexers, the only way is to use your Indexers in Active/Passive mode, manually passing from one status to the other (in fault situation) and loosing data in the meantime!
I know that it isn't a good solution but the best way to manage UDP logs is the one I described before.
There is also another possibility that It should run but I didn't try it: You could try to send UDP logs to the Search Heads (passing by the Load Balancer) and after they forward them to the Indexers; in other words using Search Heads also as Heavy Forwarders.
It should run because Search Heads usually forward their logs to the Indexers, You have only to verify the overload caused by UDP logs.
if you get data on one indexer directly which will replicate data on second indexer you continue to have a Single Point of Failure, because if this server fails, you lose your UDP logs until you change your UDP configuration and send logs to the other Indexer.
For this reason I suggest to use a LB because in this way you don't loose any log.
Think to implement the solution using Search Heads and LB or add a LB to the Indexers: this is a problem related to the UDP source and LB is the only solution.
UDP is inherently non-highly available. There is not any acknowledgement from the other side of the communication that tells you the packets were successfully received. there is not any error correction etc either. Therefore if you are using UDP you should already be aware that the protocol itself is inherently non-HA.
You'd have to use TCP if you needed HA.
To get around this most implementations of UDP send the exact same message more than once. SYSLOG is commonly configured to send the same UDP message multiple times, etc. So it really is no use trying to load balance UDP.
Just go with active/passive UDP architecture and where HA is required, demand that the customer uses you TCP architecture instead.
You need to stand up a
syslog-ng server between to receive the UDP and then use
[monitor:// to send to the indexers (DO NOT UPD directly to the indexers).