outputs.conf causing blocking when one indexer out of 6 is down for short periods

[lukessi]

Hi,

I have 6 indexers and when one or two has gone down it moans about it and blocks traffic for a few mins then when it can lb to working one it continues.. Is there a way of setting a don't use this indexer for x amount of time? The spec file is covers so many options.... I just want to know what others do?

[richgalloway]

What you describe seems unusual. Can you share the outputs.conf settings?
What is being blocked, searches or indexing?

[lukessi]

Hi its the uni fwd saying its blocking the output queue, once it finds a working indexer again it shoves all the data but the queues on my UF's are small. 

[tcpout]

defaultGroup = cluster_indexers

disabled = false



[tcpout:cluster_indexers]

#server = index1:9997,index2:9997

server = index1:9997,index2:9997,newindex01:9997,newindex02:9997,newindex03:9997,newindex04:9997

useACK = true

[richgalloway]

Consider adjusting the readTimeout and writeTimeout values in outputs.conf.  The default for each is 5 minutes.

[lukessi · Accepted solution]

That is more do with ack's coming back and forth. I don't see that its like it round robins but keeps trying ones that are down. After a little more reading I am going to use Index Discovery, so the cluster master tells the UF's what indexers are up.

https://docs.splunk.com/Documentation/Splunk/7.0.0/Indexer/indexerdiscovery

Thanks for the replies all.

[isoutamo]

Did you have a multi site cluster where one site has significantly less indexers than other sites? If so then site replication factor can cause this.

r. Ismo