I am looking for tips, documentatio, etc in order to setup data replication between 2 production Splunk servers. So, when data gets indexed on server 1, then server 2 indexes the same data. However, if server 1 goes down and server 2 is active, server 2 indexes data, then when server 1 goes back online, data gets indexed.
As of Splunk 5.0, we've introduced Index Replication to handle data replication and recover gracefully from server failures. More info can be found here
You can setup data replication using Splunk configuration(ie: your Indexers or Forwarders can clone events over to your replication Indexers) or at the infrastructure level(ie: SAN replication). Note , this answer is relevant to version 4.2/4.3 of Splunk.
Have a look at this link :