Do we need to forward to all indexers in a two site index cluster?

[Gregski11]

I took over an established Splunk ecosystem when the main support admin retired.  I noticed that not all of our stand alone Search Heads, and both Deployment Servers are setup to forward to all 12 of our Indexers in a single multi site Indexer Cluster (see table below)

Some Search Heads in their outputs.conf only list the six Indexers that are assigned to Site 1, while the other Search Heads in their outputs.conf only list the six Indexers assigned to Site 2

However all Search Heads in their server.conf file have this stanza:

[clustering]
multisite = true

So the question is should all of our Splunk Instances aka Search Heads, Cluster Master, and Deployment Servers have all 12 Indexers defined in their outputs.conf ? 

Site 1	Site 2
Indexer01	Indexer07
Indexer02	Indexer08
Indexer03	Indexer09
Indexer04	Indexer10
Indexer05	Indexer11
Indexer06	Indexer12

[scelikok]

Hi @Gregski11,

As a general best practise, the nodes that have site = site1 in [general] stanza in server.conf  , should have Site1 indexers in their outputs.conf  . And vice versa.  This will prevent log traffic across sites.

If you want a failover capability for forwarding logs you can check the indexer discovery on link below;

Use indexer discovery in a multisite cluster

[PickleRick]

Should is not a good word to ask about because every environment is different and maybe in some of them you can some options are better than other.

But in general, there's no requirement to direct your outputs to one site or another or both of them. Theoretically, replication should take care about it. That's why you have site SF and site RF - to make the cluster replicate data across the nodes.

It all depends on the distribution of your sources and your sites parameters.

For example, if you have two sites with site SF and RF set as site1:1,site2:1,origin:2, you might want your forwarders to spread the load across sites, otherwise one of the sites will hold more data than the other.

If you had half of your forwarders in site1 and pointing at site1 and another half in site2 pointed at site2, that could be OK but if they were all in site1 and pointed only at site1, you'd end up with site1 holding twice the data of site2 which might not be desired.

So there's no single "right" answer here. It all depends on your requirements and circumstances.

EDIT: Oh, I just noticed you were talking about internal logs from the Splunk environment itself - probably the volume of data won't be that significant compared to the "production" data you'll be ingesting so you might get away with pushing them only to "local" site and it won't matter that much. But again - depends on your requirements for data availability. And yes, I agree with @gcusello that while Community can give you some general advise, for detailed solution to such topic it's best to employ an Architect. 🙂

[gcusello]

Hi @Gregski11,

sorry, but a multi site clustered architecture isn't a question for the Community: you need a Splunk Architect or a Splunk Professional Service!

Anyway, see "Affinity" in a Splunk architecture (e.g. https://docs.splunk.com/Documentation/Splunk/9.0.3/Indexer/Multisitesearchaffinity )to answer to your question.

In few words, usually Splunk internal logs are sent to the near indexes, and they reply to the other site Indexers.

Ciao.

Giuseppe

[Gregski11]

thanks Rich I think this link is more my speed, now just to interpret it from Greek to English, lol

Managing Indexers and Clusters of Indexers 

Configure multi-cluster search for multisite indexer clusters

A search head can search across multiple multisite clusters or a combination of single-site and multisite clusters. To configure this, you need to specify the search head's site attribute when connecting it to a multisite cluster.

By editing server.conf

To configure multi-cluster search for a multisite cluster, you need to set two multisite-specific attributes: site and multisite. The locations of these attributes vary, depending on a few factors.

If the search head will be searching across only multisite clusters, and the search head is on the same site in each cluster, put the site attribute under the [general] stanza and the multisite attribute under each [clustermanager] stanza:

If the search head will be searching across only multisite clusters, and the search head is on a different site in each cluster, put both the site and the multisite attributes under the [clustermanager] stanzas:

If the search head will be searching across a combination of single-site and multisite clusters, put both the site and the multisite attributes under the [clustermanager] stanza for any multisite clusters. In this example, the search head searches across two clusters, only one of which is multisite:

[PickleRick]

OK. But this is about searching, not about forwarding events to indexers and you asked about forwarding. So maybe you asked wrong question 😉