Deployment Architecture

Splunk Index Clustering

Kendo213
Communicator

We have a multi-site cluster of 6.6.4 (site1, site2) and when testing a downed state of site1, some of our forwarders were not failing to the site that was up. Or, we saw odd behavior like forwarders still attempting to hit site1 AND site2 (almost like a round robin) indexers despite the cluster master noting them as unknown/down.

Here is what our outputs.conf looks like:

[tcpout]
defaultGroup = primary_indexers
maxQueueSize = 7MB
useACK = true
forceTimebasedAutoLB = true
forwardedindex.2.whitelist = (_audit|_introspection|_internal)

[tcpout:primary_indexers]
indexerDiscovery = clustered_indexers

sslCertPath = $SPLUNK_HOME/etc/apps/forwarderoutputs/local/splunkforwarder.pem
sslPassword = pass
sslVerifyServerCert = true

[indexer_discovery:clustered_indexers]
pass4SymmKey = pass
master_uri = https://clustermaster:8089

[tcpout-server://site01index01:9998]
sslVerifyServerCert = true
sslAltNameToCheck = site01index01, site01index01.domain.com

[tcpout-server://site01index02:9998]
sslVerifyServerCert = true
sslAltNameToCheck = site01index02, site01index02.domain.com

[tcpout-server://site2index01:9998]
sslVerifyServerCert = true
sslAltNameToCheck = site02index01, site02index01.domain.com

[tcpout-server://site2index02:9998]
sslVerifyServerCert = true
sslAltNameToCheck = site02index02, site02index02.domain.com

0 Karma
1 Solution

FrankVl
Ultra Champion

Isn't that a result of specifying those tcpout-server stanzas? As far as I understand, those are not necessary when using indexer discovery. Why do you have those?

View solution in original post

FrankVl
Ultra Champion

Isn't that a result of specifying those tcpout-server stanzas? As far as I understand, those are not necessary when using indexer discovery. Why do you have those?

View solution in original post

Kendo213
Communicator

That's what I was thinking, however this is how Splunk PS configured it.

0 Karma

amitm05
Builder

Can you state your question more clearly please ?

0 Karma

Kendo213
Communicator

I guess I would expect my forwarders to hit the cluster master, and be told that "these indexers are down" and start forwarding to the indexers that are up according to the cluster master. That doesn't seem to be happening

0 Karma