Splunk Index Clustering

[Kendo213]

We have a multi-site cluster of 6.6.4 (site1, site2) and when testing a downed state of site1, some of our forwarders were not failing to the site that was up. Or, we saw odd behavior like forwarders still attempting to hit site1 AND site2 (almost like a round robin) indexers despite the cluster master noting them as unknown/down.

Here is what our outputs.conf looks like:

[tcpout]
defaultGroup = primary_indexers
maxQueueSize = 7MB
useACK = true
forceTimebasedAutoLB = true
forwardedindex.2.whitelist = (_audit|_introspection|_internal)

[tcpout:primary_indexers]
indexerDiscovery = clustered_indexers

sslCertPath = $SPLUNK_HOME/etc/apps/forwarderoutputs/local/splunkforwarder.pem
sslPassword = pass
sslVerifyServerCert = true

[indexer_discovery:clustered_indexers]
pass4SymmKey = pass
master_uri = https://clustermaster:8089

[tcpout-server://site01index01:9998]
sslVerifyServerCert = true
sslAltNameToCheck = site01index01, site01index01.domain.com

[tcpout-server://site01index02:9998]
sslVerifyServerCert = true
sslAltNameToCheck = site01index02, site01index02.domain.com

[tcpout-server://site2index01:9998]
sslVerifyServerCert = true
sslAltNameToCheck = site02index01, site02index01.domain.com

[tcpout-server://site2index02:9998]
sslVerifyServerCert = true
sslAltNameToCheck = site02index02, site02index02.domain.com

[FrankVl · Accepted solution]

Isn't that a result of specifying those tcpout-server stanzas? As far as I understand, those are not necessary when using indexer discovery. Why do you have those?

[Kendo213]

That's what I was thinking, however this is how Splunk PS configured it.

[amitm05]

Can you state your question more clearly please ?

[Kendo213]

I guess I would expect my forwarders to hit the cluster master, and be told that "these indexers are down" and start forwarding to the indexers that are up according to the cluster master. That doesn't seem to be happening