When an index is been replicated and the source node in question loses connectivity to the cluster, to the target node in question, how does the bucket get recovered/continued index replication when connectivity is back?
Does Splunk keep a marker or something to the exact event it was on? I don't know how to articulate this.
Any help would be appreciated.
I would like to setup three "satellite" nodes in a cluster that replicate one index to the cluster but they themselves do not receive any replication information as they are on WAN links and can be down at times.
It's hard to answer because there's a lot of scenarios and I don't know what "source" and "target" mean here.
Are we discussing a case where a forwarder F is sending data to indexer I1 which is part of cluster CL that also contains indexers I2 and I3, and I1 loses communication with I2 and I3?
In this case, Indexer I1 does not acknowlege (ack) the data it is getting from F until I2 and I3 have accepted the data. This means that after I1 loses communication with the rest of the cluster, it would not be able to tell forwarder F that it has taken ownership of the data until it rejoins the cluster.
Thus, the data would remain owned by the forwarder F and the logfiles or other sources it is collecting data from.
However, the idea that the indexers I1, I2, and I3 lose connectivity should be a rare event. There are a variety of scenarios where intermittent or broken connectivity in the 5.0 clustering design will not behave well. Typically this manifests as throughput problems, but I'm not an expert. The currently shipped code is more or less designed for systems with relatively reliable fast communication paths. There's talk of building for multiple sites in the future.
Our clustering is system that replications components of an index, specifically buckets. Thus the forwarding doesn't really participate. It just knows it has to send its data to the indexers it's supposed to send it to. Typically it would be configured to send to any of the three.
I was thinking forwarder F is part of the cluster and in order to overcome index and forward licence costs, it replicates an index upstream to nodes I1, I2. These two indexers then maintain the replication and search factor of two. Forwarder F, apart from replication upstream of said index doesn't keep any other searchable or non-searchable indexes on said localhost. So forwarder F is the indexer on site at the client behind a WAN link and it shouldn't receive any replication data as a cluster member. This is not possible out the box I know.