Deployment Architecture

Splunk Index Clustering

Kendo213
Communicator

We have a multi-site cluster of 6.6.4 (site1, site2) and when testing a downed state of site1, some of our forwarders were not failing to the site that was up. Or, we saw odd behavior like forwarders still attempting to hit site1 AND site2 (almost like a round robin) indexers despite the cluster master noting them as unknown/down.

Here is what our outputs.conf looks like:

[tcpout]
defaultGroup = primary_indexers
maxQueueSize = 7MB
useACK = true
forceTimebasedAutoLB = true
forwardedindex.2.whitelist = (_audit|_introspection|_internal)

[tcpout:primary_indexers]
indexerDiscovery = clustered_indexers

sslCertPath = $SPLUNK_HOME/etc/apps/forwarderoutputs/local/splunkforwarder.pem
sslPassword = pass
sslVerifyServerCert = true

[indexer_discovery:clustered_indexers]
pass4SymmKey = pass
master_uri = https://clustermaster:8089

[tcpout-server://site01index01:9998]
sslVerifyServerCert = true
sslAltNameToCheck = site01index01, site01index01.domain.com

[tcpout-server://site01index02:9998]
sslVerifyServerCert = true
sslAltNameToCheck = site01index02, site01index02.domain.com

[tcpout-server://site2index01:9998]
sslVerifyServerCert = true
sslAltNameToCheck = site02index01, site02index01.domain.com

[tcpout-server://site2index02:9998]
sslVerifyServerCert = true
sslAltNameToCheck = site02index02, site02index02.domain.com

0 Karma
1 Solution

FrankVl
Ultra Champion

Isn't that a result of specifying those tcpout-server stanzas? As far as I understand, those are not necessary when using indexer discovery. Why do you have those?

View solution in original post

FrankVl
Ultra Champion

Isn't that a result of specifying those tcpout-server stanzas? As far as I understand, those are not necessary when using indexer discovery. Why do you have those?

Kendo213
Communicator

That's what I was thinking, however this is how Splunk PS configured it.

0 Karma

amitm05
Builder

Can you state your question more clearly please ?

0 Karma

Kendo213
Communicator

I guess I would expect my forwarders to hit the cluster master, and be told that "these indexers are down" and start forwarding to the indexers that are up according to the cluster master. That doesn't seem to be happening

0 Karma
Get Updates on the Splunk Community!

What You Read The Most: Splunk Lantern’s Most Popular Articles!

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Announcing the General Availability of Splunk Enterprise Security 8.1!

We are pleased to announce the general availability of Splunk Enterprise Security 8.1. Splunk becomes the only ...

Developer Spotlight with William Searle

The Splunk Guy: A Developer’s Path from Web to Cloud William is a Splunk Professional Services Consultant with ...