Deployment Architecture

[<indexer hostname>] Timed out waiting for peer <indexer hostname>:ingest_pipe=1


We are using splunk version 8.1.0 in cluster mode , in my environment we have this components:
Nginx load load balancer : for load balancing request to search heads
3 search heads and 1 deployer: in cluster mode
3 indexer and 1 master node: in cluster mode
2 heavy forwarder : stand alone and forward data with load balancing between indexers
1 syslog server : receive syslogs from 100 servers and send it via ipvsadm(port 514 udp) to heavy forwarders
All splunk servers is centos 7 and all servers are same network zone
And i have almost 300 GB per day data
server specifications:
Search heads : 32GB Ram 32Core Cpu
Indexer : 32GB Ram 16Core Cpu
heavy forwarder : 12GB Ram 12Core Cpu
syslog server: 12GB Ram 12Core Cpu

We have a problem in real time search , we have a lot of dashboards with multiple searches in there , when i open my dashboards after random time (about 1 to 120 seconds) we get a error.
here is description of my error : [<indexer hostname>] Timed out waiting for peer <indexer hostname>:ingest_pipe=1. Search results might be incomplete! If this occurs frequently, receiveTimeout in distsearch.conf might need to be increased
we dont have any problem in resources such as cpu utilization and lack of memory too

This error happened while we have another instance with one indexer and one search head in non cluster environment with same traffic, and we dont have any problem with that , the old version of splunk is 6.6.1
So what did i do:
- Increase receiveTimeout parameter in search heads but i know problem is not this
- Increase parallelIngestionPipelines in indexers to 2 ,
- Tune os recommended by splunk site
- Increase max_searches_per_cpu to 15
- and ...
But problem not solved

0 Karma
Get Updates on the Splunk Community!

New Splunk Observability innovations: Deeper visibility and smarter alerting to ...

You asked, we delivered. Splunk Observability Cloud has several new innovations giving you deeper visibility ...

Synthetic Monitoring: Not your Grandma’s Polyester! Tech Talk: DevOps Edition

Register today and join TekStream on Tuesday, February 28 at 11am PT/2pm ET for a demonstration of Splunk ...

Instrumenting Java Websocket Messaging

Instrumenting Java Websocket MessagingThis article is a code-based discussion of passing OpenTelemetry trace ...