I have a scenario and was wondering if somebody could confirm what would happen...
Lets pretend we're the Federation, obviously we have a lot of data across the galaxy but we're really quite interested in our local star system.
To that end we've decided to install Splunk v5 (they obtained it after a time travelling related accident). So on the Enterprise we install an indexer, Excelsior also has an Indexer, Spacedock (in orbit of earth) has two indexers (its generating quite a lot of data).
Finally, on the moon we have another two indexers.
On earth we have a couple of search heads dotted around. Because we've installed v5 we decide to to setup a cluster to make sure we have HA across the fleet, the master is located on a dedicated terminal on the Spacedock.
All works well for a week when suddenly, a giant cylindrical thing with a giant floating ball and crazy whale noises appears, no one has any idea whats going on when suddenly Spacedock loses all power.
Our master and two indexers are taken offline.
According to the docs the search heads will continue to try and search across its previously known indexers, so in this case HA has actually failed and we don't have any redundancy from entire site failures if the master is located on that site. Is that correct? Is there any way to mitigate or protect against this? (Short of sticking the master on a satellite)
Thanks for any opinions or views, the more the merrier.
