Thanks for your comments.
The peer is working when we open one screen, but when we increase it to 5 to do a load test we get the message. All machines are on 56 Core with lots of RAM.
A heave screen is one that runs 40 searches when it is opened. However, 20 are run in less than 1 seconds and about 5 take 5 seconds to complete normally.
This doesn't happen on standalone instances because they don't use distributed search.
Have you checked the indexer at 10.25.57.21? Is it up and listening on port 8089? Is a firewall blocking communications to that address/port? Does the indexer run low on resources when processing 5 heavy screens? What is a "heavy screen"?
Thanks for your questions.
Yes the peer is up on 10.25.57.21 and is working when we load in a screen on its own, and it is fast - just like production. A heavy screen can load 40 searches in parallel. 20 are finished in 1 second and about 5 take 20 seconds when loaded as a single screen and not part of a load test (load test = 5 screens in parallel).
The CPU and RAM on the INDEXER do not move much, the network.
when we open up other screens we are getting "Waiting for queued job to start" - but we have given this user a lot of capacity...
Below is the network activity before and after the test. I am not sure if this is ok or not