1.Do I need more Splunk instances?
Ideally Yes. A search head cluster requires a minimum of 3 members. So with just two search heads you will not be able to have a search head cluster.
Also for indexers, your options depend on your need for resiliency, the cluster can tolerate a failure of (replication factor - 1) peer nodes. And there are benefits to having a separate cluster master for your indexers.
Search head cluster will need a deployer as well. You do have the option of having a server take on multiple roles... still if your preference is to have both a search head and an Indexer cluster you will need more servers for a clean deployment.
2.Do I need to send syslog to only one indexer, or the same syslog to two indexers?
To get the maximum performance benefit it's preferable to send your data to all indexers (distribute the data). So when the searches hit the indexers, each peer node can process it's set of results and render them back much faster.
3.If I send data to only one indexer, with replication, will I have the same data in two indexers?
Yes. Else when an indexer goes down, your data is lost. Clustering requires indexers maintaining replicated copies of the data (as defined by the replication factor)
4.If I send same data to two indexers, with replication, will I have data copies twice, in two indexers?
I am assuming you mean, you want to send the data to two indexers and not send the "same" data (clone) to two indexers. You can also have the option of using an intermediate forwarder which will load balance the data for you. But yes, depending on how you will be sending the data there is the possibility of having duplicates ... which will get replicated.
5.If one indexer is down, will the other one be enough for service continuity?
Yes. If you enable clustering, a replication factor of 2 will ensure availability even on failure of 1 peer node.
6.If I have a traffic balancer, only for sending syslog data, can I send data to any indexer, do I need any special consideration?
Sending data to any indexer is fine... though as explained it’s better to distribute the data.
Additionally there is excellent documentation on clustering. Check out Indexer Clustering and Search Head Clustering.
Good luck with the deployment.
... View more