Deployment Architecture

[<indexer hostname>] Timed out waiting for peer <indexer hostname>:ingest_pipe=1


We are using splunk version 8.1.0 in cluster mode , in my environment we have this components:
Nginx load load balancer : for load balancing request to search heads
3 search heads and 1 deployer: in cluster mode
3 indexer and 1 master node: in cluster mode
2 heavy forwarder : stand alone and forward data with load balancing between indexers
1 syslog server : receive syslogs from 100 servers and send it via ipvsadm(port 514 udp) to heavy forwarders
All splunk servers is centos 7 and all servers are same network zone
And i have almost 300 GB per day data
server specifications:
Search heads : 32GB Ram 32Core Cpu
Indexer : 32GB Ram 16Core Cpu
heavy forwarder : 12GB Ram 12Core Cpu
syslog server: 12GB Ram 12Core Cpu

We have a problem in real time search , we have a lot of dashboards with multiple searches in there , when i open my dashboards after random time (about 1 to 120 seconds) we get a error.
here is description of my error : [<indexer hostname>] Timed out waiting for peer <indexer hostname>:ingest_pipe=1. Search results might be incomplete! If this occurs frequently, receiveTimeout in distsearch.conf might need to be increased
we dont have any problem in resources such as cpu utilization and lack of memory too

This error happened while we have another instance with one indexer and one search head in non cluster environment with same traffic, and we dont have any problem with that , the old version of splunk is 6.6.1
So what did i do:
- Increase receiveTimeout parameter in search heads but i know problem is not this
- Increase parallelIngestionPipelines in indexers to 2 ,
- Tune os recommended by splunk site
- Increase max_searches_per_cpu to 15
- and ...
But problem not solved

0 Karma
Get Updates on the Splunk Community!

Splunk Education - Fast Start Program!

Welcome to Splunk Education! Splunk training programs are designed to enable you to get started quickly and ...

Five Subtly Different Ways of Adding Manual Instrumentation in Java

You can find the code of this example on GitHub here. Please feel free to star the repository to keep in ...

New Splunk APM Enhancements Help Troubleshoot Your MySQL and NoSQL Databases Faster

Splunk Observability has two new enhancements to make it quicker and easier to troubleshoot slow or frequently ...