Splunk Search

Unknown error for indexer: XXXX . Search Results might be incomplete! If this occurs frequently, check on

robertlynch2020
Motivator
Hi
 
I am migrating from a single install to a cluster 1SH + 1MD + 3 Indexers.
 
When we are trying a load test - 5 heavy screens in parallel we are getting the following errors - this was not the case in the signal install and we think perhaps we are missing a prop?
 
[subsearch]: Unknown error for indexer: hp925srv_INDEXER4. Search Results might be incomplete! If this occurs frequently, check on the peer.
 
Unable to distribute to peer named 10.25.57.21:8089 at uri=10.25.57.21:8089 using the uri-scheme=https because peer has status=Down. Verify uri-scheme, connectivity to the search peer, that the search peer is up, and that an adequate level of system resources are available. See the Troubleshooting Manual for more information.
 
[subsearch]: Error connecting: Connect Timeout

Regards

Robert

Labels (1)
Tags (1)
0 Karma

robertlynch2020
Motivator

Hi 

Thanks for your comments.

The peer is working when we open one screen, but when we increase it to 5 to do a load test we get the message. All machines are on 56 Core with lots of RAM. 

A heave screen is one that runs 40 searches when it is opened. However, 20 are run in less than 1 seconds and about 5 take 5 seconds to complete normally.

Regards

Robert

0 Karma

soutamo
SplunkTrust
SplunkTrust
What kind of disk subsystem you have on those nodes and how many IOPS it gives to you?
0 Karma

robertlynch2020
Motivator

Hi - Thanks for the question.

We are 11TB of SSD on each node and no subsystem.

I am assuming this is ok and should give me the performance I need?

Rob

0 Karma

richgalloway
SplunkTrust
SplunkTrust

This doesn't happen on standalone instances because they don't use distributed search.

Have you checked the indexer at 10.25.57.21?  Is it up and listening on port 8089?  Is a firewall blocking communications to that address/port?  Does the indexer run low on resources when processing 5 heavy screens?  What is a "heavy screen"?

---
If this reply helps you, an upvote would be appreciated.
0 Karma

robertlynch2020
Motivator

Hi

Thanks for your questions.

Yes the peer is up on 10.25.57.21 and is working when we load in a screen on its own, and it is fast - just like production. A heavy screen can load 40 searches in parallel. 20 are finished in 1 second and about 5 take 20 seconds when loaded as a single screen and not part of a load test (load test = 5 screens in parallel).

The CPU and RAM on the INDEXER do not move much, the network.

when we open up other screens we are getting  "Waiting for queued job to start" - but we have given this user a lot of capacity...

Below is the network activity before and after the test. I am not sure if this is ok or not

2020-10-30 10_48_40-Window.png

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!