Splunk Enterprise

outputs.conf causing blocking when one indexer out of 6 is down for short periods

lukessi
Path Finder

Hi,

I have 6 indexers and when one or two has gone down it moans about it and blocks traffic for a few mins then when it can lb to working one it continues.. Is there a way of setting a don't use this indexer for x amount of time? The spec file is covers so many options.... I just want to know what others do?

Labels (2)
0 Karma
1 Solution

lukessi
Path Finder

That is more do with ack's coming back and forth. I don't see that its like it round robins but keeps trying ones that are down. After a little more reading I am going to use Index Discovery, so the cluster master tells the UF's what indexers are up.

https://docs.splunk.com/Documentation/Splunk/7.0.0/Indexer/indexerdiscovery

 

Thanks for the replies all.

View solution in original post

0 Karma

richgalloway
SplunkTrust
SplunkTrust
What you describe seems unusual. Can you share the outputs.conf settings?
What is being blocked, searches or indexing?
---
If this reply helps you, Karma would be appreciated.
0 Karma

lukessi
Path Finder

Hi its the uni fwd saying its blocking the output queue, once it finds a working indexer again it shoves all the data but the queues on my UF's are small. 

 

 

[tcpout]

defaultGroup = cluster_indexers

disabled = false



[tcpout:cluster_indexers]

#server = index1:9997,index2:9997

server = index1:9997,index2:9997,newindex01:9997,newindex02:9997,newindex03:9997,newindex04:9997

useACK = true

 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Consider adjusting the readTimeout and writeTimeout values in outputs.conf.  The default for each is 5 minutes.

---
If this reply helps you, Karma would be appreciated.
0 Karma

lukessi
Path Finder

That is more do with ack's coming back and forth. I don't see that its like it round robins but keeps trying ones that are down. After a little more reading I am going to use Index Discovery, so the cluster master tells the UF's what indexers are up.

https://docs.splunk.com/Documentation/Splunk/7.0.0/Indexer/indexerdiscovery

 

Thanks for the replies all.

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Did you have a multi site cluster where one site has significantly less indexers than other sites? If so then site replication factor can cause this.

r. Ismo

0 Karma
Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...