I am looking for tips, documentatio, etc in order to setup data replication between 2 production Splunk servers. So, when data gets indexed on server 1, then server 2 indexes the same data. However, if server 1 goes down and server 2 is active, server 2 indexes data, then when server 1 goes back online, data gets indexed.
As of Splunk 5.0, we've introduced Index Replication to handle data replication and recover gracefully from server failures. More info can be found here
http://docs.splunk.com/Documentation/Splunk/5.0/Indexer/Aboutclusters
Does this work for 2 servers only? i.e. both servers acting as indexers and search heads?
Look in to the current Splunk beta documentation. It will do what you are looking for in a clustered state.
You can setup data replication using Splunk configuration(ie: your Indexers or Forwarders can clone events over to your replication Indexers) or at the infrastructure level(ie: SAN replication). Note , this answer is relevant to version 4.2/4.3 of Splunk.
Have a look at this link :
http://docs.splunk.com/Documentation/Splunk/4.3/Installation/Highavailabilityreferencearchitecture#D...
there are links from that topic to configuration details, for example: http://docs.splunk.com/Documentation/Splunk/4.3/Deploy/Setuploadbalancingd
Ok. I read and now I know that I can do that, but where may I see some config configuration examples? I cannot find anything about that.