topic Re: outputs.conf causing blocking when one indexer out of 6 is down for short periods in Splunk Enterprise

outputs.conf causing blocking when one indexer out of 6 is down for short periods

lukessi — Wed, 08 Jul 2020 14:14:48 GMT

Hi,

I have 6 indexers and when one or two has gone down it moans about it and blocks traffic for a few mins then when it can lb to working one it continues.. Is there a way of setting a don't use this indexer for x amount of time? The spec file is covers so many options.... I just want to know what others do?

Re: outputs.conf causing blocking when one indexer out of 6 is down for short periods

richgalloway — Wed, 08 Jul 2020 15:12:56 GMT

What you describe seems unusual. Can you share the outputs.conf settings?
What is being blocked, searches or indexing?

Re: outputs.conf causing blocking when one indexer out of 6 is down for short periods

lukessi — Wed, 08 Jul 2020 15:21:56 GMT

Hi its the uni fwd saying its blocking the output queue, once it finds a working indexer again it shoves all the data but the queues on my UF's are small.

[tcpout] defaultGroup = cluster_indexers disabled = false [tcpout:cluster_indexers] #server = index1:9997,index2:9997 server = index1:9997,index2:9997,newindex01:9997,newindex02:9997,newindex03:9997,newindex04:9997 useACK = true

Re: outputs.conf causing blocking when one indexer out of 6 is down for short periods

isoutamo — Wed, 08 Jul 2020 18:41:56 GMT

Did you have a multi site cluster where one site has significantly less indexers than other sites? If so then site replication factor can cause this.

r. Ismo

Re: outputs.conf causing blocking when one indexer out of 6 is down for short periods

richgalloway — Wed, 08 Jul 2020 20:01:53 GMT

Consider adjusting the readTimeout and writeTimeout values in outputs.conf. The default for each is 5 minutes.

Re: outputs.conf causing blocking when one indexer out of 6 is down for short periods

lukessi — Thu, 09 Jul 2020 07:45:46 GMT

That is more do with ack's coming back and forth. I don't see that its like it round robins but keeps trying ones that are down. After a little more reading I am going to use Index Discovery, so the cluster master tells the UF's what indexers are up.

https://docs.splunk.com/Documentation/Splunk/7.0.0/Indexer/indexerdiscovery

Thanks for the replies all.