optimizing Splunk multisite indexer replication traffic routes

[satyenshah]

Is there a combination of settings that can optimize replication in a multisite cluster with 3+ sites where the links between sites are not identical?

Suppose you have three sites (site1, site2, site3).  Suppose site1 and site2 are connected by a fast link, and that site3 is far away with a slow link.  Suppose a bucket exists on site2 and site3 but not site1.

I there a combination of settings or hacks that can ensure site1 gets the bucket from nearby site2 and not from distant site3?

[henry_collins]

In a multisite cluster, Splunk’s replication placement logic doesn’t really optimize based on link speed or latency. It follows the site replication or search factor rules and the bucket copies available, not network topology awareness. The usual approach is to design the site configuration so copies are distributed in a way that matches your network layout. If you have very uneven links, you may need to rethink the site architecture rather than rely on Splunk to pick the “closest” peer.

There isn’t a simple setting that forces site1 to always pull from the fastest site2 peer instead of site3. Network optimization usually comes from proper capacity planning, site definitions, and ensuring the desired copies exist where you need them.

[satyenshah]

Of course there's no simple setting, but the question is if there's a combination of settings (e.g. timeouts, retries,  retry intervals, iptables) which can mimic that absent setting.

For example, I've tried simply blocking port 8080 traffic from distant site3 to site2, to see if a recently-rebuilt peer in site2 would be able to fixup exclusively from site1, by exhausting retries for replicating a bucket from site3 until finally giving up, then retrying from site1, and then succeeding.  That apparently throws errors that jam up the fixup tasks.  But, with clever settings, it might work.

[PickleRick]

I'm not sure the CM's policy of assigning replication partners is anywhere officially documented so the best you could hope for would be trial and error. Anyway, if you deliberately broke replication from site1 to site2 in hope of getting buckets to go from site3 to site2... You'd get a lot of errors and alerts. I suppose your CM would have to do much more work because initial replication attempts would fail and it would have to initiate fixups (which could fail again.

So I would strongly advise against going that route. At best you'd have an unsupported setup with a very sensitive configuration which could go sideways at any moment. At worst - you'd break your whole clustering setup. And no sane support engineer will help you here other than by saying "go to the default settings".