I am looking for tips, documentatio, etc in order to setup data replication between 2 production Splunk servers. So, when data gets indexed on server 1, then server 2 indexes the same data. However, if server 1 goes down and server 2 is active, server 2 indexes data, then when server 1 goes back online, data gets indexed.
You can setup data replication using Splunk configuration(ie: your Indexers or Forwarders can clone events over to your replication Indexers) or at the infrastructure level(ie: SAN replication). Note , this answer is relevant to version 4.2/4.3 of Splunk.
Have a look at this link :
As of Splunk 5.0, we've introduced Index Replication to handle data replication and recover gracefully from server failures. More info can be found here