Hi
We are using splunk version 8.1.0 in cluster mode , in my environment we have this components:
Nginx load load balancer : for load balancing request to search heads
3 search heads and 1 deployer: in cluster mode
3 indexer and 1 master node: in cluster mode
2 heavy forwarder : stand alone and forward data with load balancing between indexers
1 syslog server : receive syslogs from 100 servers and send it via ipvsadm(port 514 udp) to heavy forwarders
All splunk servers is centos 7 and all servers are same network zone
And i have almost 300 GB per day data
server specifications:
Search heads : 32GB Ram 32Core Cpu
Indexer : 32GB Ram 16Core Cpu
heavy forwarder : 12GB Ram 12Core Cpu
syslog server: 12GB Ram 12Core Cpu
We have a problem in real time search , we have a lot of dashboards with multiple searches in there , when i open my dashboards after random time (about 1 to 120 seconds) we get a error.
here is description of my error : [<indexer hostname>] Timed out waiting for peer <indexer hostname>:ingest_pipe=1. Search results might be incomplete! If this occurs frequently, receiveTimeout in distsearch.conf might need to be increased
we dont have any problem in resources such as cpu utilization and lack of memory too
This error happened while we have another instance with one indexer and one search head in non cluster environment with same traffic, and we dont have any problem with that , the old version of splunk is 6.6.1
So what did i do:
- Increase receiveTimeout parameter in search heads but i know problem is not this
- Increase parallelIngestionPipelines in indexers to 2 ,
- Tune os recommended by splunk site
- Increase max_searches_per_cpu to 15
- and ...
But problem not solved
