Getting Data In

When adding an indexer to a distributed environment, is there a configuration that makes indexers exchange events to auto load balance them?

adamguzek
Explorer

Is there a configuration that makes indexers exchange events in order to auto load balance them? Let's say I add an indexer into distributed environment. I want to use it without reconfiguring syslog sources and forwarders.

Maybe it's a request - make indexers connect to each other, and move events between them to distribute in an optimal way...

Does indexer clustering with duplication of data give any advantage? Maybe then the search head is using first/second indexer to retrieve events... Not only "first copy"?

1 Solution

dwaddle
SplunkTrust
SplunkTrust

In a distributed, non-clustered, environment the answer is a resounding 'no'. The various indexers have no knowledge of each other, there is no shared state across indexers. Both the search heads and the forwarders must be given knowledge of all the indexers. If you add an indexer and only configure it into your search head for distributed search, then it will get no data at all. You can't feed it data without making changes to your forwarders to send data to it.

When you enable clustering, this gives the indexer peers knowledge of each other, but only for the purpose of making redundant copies. An indexer can make an additional copy of data at a peer, but it cannot "migrate" its data to that peer. Come search time, an indexer bucket has but one primary copy, and it is only the primary copy that is searched. Any additional secondary copies do not participate in the search.

For the most part, the requirement that the forwarders know about all indexers does not change when you enable clustering. But, as of Splunk 6.3, the indexer discovery feature allows for forwarders to contact a cluster master and simply ask it "what indexers should I connect to?" Then when you add new indexers to the cluster, the forwarders learn of them automatically.

View solution in original post

dwaddle
SplunkTrust
SplunkTrust

In a distributed, non-clustered, environment the answer is a resounding 'no'. The various indexers have no knowledge of each other, there is no shared state across indexers. Both the search heads and the forwarders must be given knowledge of all the indexers. If you add an indexer and only configure it into your search head for distributed search, then it will get no data at all. You can't feed it data without making changes to your forwarders to send data to it.

When you enable clustering, this gives the indexer peers knowledge of each other, but only for the purpose of making redundant copies. An indexer can make an additional copy of data at a peer, but it cannot "migrate" its data to that peer. Come search time, an indexer bucket has but one primary copy, and it is only the primary copy that is searched. Any additional secondary copies do not participate in the search.

For the most part, the requirement that the forwarders know about all indexers does not change when you enable clustering. But, as of Splunk 6.3, the indexer discovery feature allows for forwarders to contact a cluster master and simply ask it "what indexers should I connect to?" Then when you add new indexers to the cluster, the forwarders learn of them automatically.

Get Updates on the Splunk Community!

What's New in Splunk Enterprise 9.4: Features to Power Your Digital Resilience

Hey Splunky People! We are excited to share the latest updates in Splunk Enterprise 9.4. In this release we ...

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

SignalFlow: What? Why? How?

What is SignalFlow? Splunk Observability Cloud’s analytics engine, SignalFlow, opens up a world of in-depth ...