topic Splunk Index Clustering in Deployment Architecture

Splunk Index Clustering

Kendo213 — Tue, 29 Sep 2020 19:18:58 GMT

We have a multi-site cluster of 6.6.4 (site1, site2) and when testing a downed state of site1, some of our forwarders were not failing to the site that was up. Or, we saw odd behavior like forwarders still attempting to hit site1 AND site2 (almost like a round robin) indexers despite the cluster master noting them as unknown/down.

Here is what our outputs.conf looks like:

[tcpout]
defaultGroup = primary_indexers
maxQueueSize = 7MB
useACK = true
forceTimebasedAutoLB = true
forwardedindex.2.whitelist = (_audit|_introspection|_internal)

[tcpout:primary_indexers]
indexerDiscovery = clustered_indexers

sslCertPath = $SPLUNK_HOME/etc/apps/forwarderoutputs/local/splunkforwarder.pem
sslPassword = pass
sslVerifyServerCert = true

[indexer_discovery:clustered_indexers]
pass4SymmKey = pass
master_uri = https://clustermaster:8089

[tcpout-server://site01index01:9998]
sslVerifyServerCert = true
sslAltNameToCheck = site01index01, site01index01.domain.com

[tcpout-server://site01index02:9998]
sslVerifyServerCert = true
sslAltNameToCheck = site01index02, site01index02.domain.com

[tcpout-server://site2index01:9998]
sslVerifyServerCert = true
sslAltNameToCheck = site02index01, site02index01.domain.com

[tcpout-server://site2index02:9998]
sslVerifyServerCert = true
sslAltNameToCheck = site02index02, site02index02.domain.com

Re: Splunk Index Clustering

amitm05 — Mon, 30 Apr 2018 15:03:34 GMT

Can you state your question more clearly please ?

Re: Splunk Index Clustering

Kendo213 — Mon, 30 Apr 2018 15:09:44 GMT

I guess I would expect my forwarders to hit the cluster master, and be told that "these indexers are down" and start forwarding to the indexers that are up according to the cluster master. That doesn't seem to be happening

Re: Splunk Index Clustering

FrankVl — Mon, 30 Apr 2018 15:32:19 GMT

Isn't that a result of specifying those tcpout-server stanzas? As far as I understand, those are not necessary when using indexer discovery. Why do you have those?

Re: Splunk Index Clustering

Kendo213 — Mon, 30 Apr 2018 15:41:47 GMT

That's what I was thinking, however this is how Splunk PS configured it.