Getting Data In

When adding an indexer to a distributed environment, is there a configuration that makes indexers exchange events to auto load balance them?

adamguzek
Explorer

Is there a configuration that makes indexers exchange events in order to auto load balance them? Let's say I add an indexer into distributed environment. I want to use it without reconfiguring syslog sources and forwarders.

Maybe it's a request - make indexers connect to each other, and move events between them to distribute in an optimal way...

Does indexer clustering with duplication of data give any advantage? Maybe then the search head is using first/second indexer to retrieve events... Not only "first copy"?

1 Solution

dwaddle
SplunkTrust
SplunkTrust

In a distributed, non-clustered, environment the answer is a resounding 'no'. The various indexers have no knowledge of each other, there is no shared state across indexers. Both the search heads and the forwarders must be given knowledge of all the indexers. If you add an indexer and only configure it into your search head for distributed search, then it will get no data at all. You can't feed it data without making changes to your forwarders to send data to it.

When you enable clustering, this gives the indexer peers knowledge of each other, but only for the purpose of making redundant copies. An indexer can make an additional copy of data at a peer, but it cannot "migrate" its data to that peer. Come search time, an indexer bucket has but one primary copy, and it is only the primary copy that is searched. Any additional secondary copies do not participate in the search.

For the most part, the requirement that the forwarders know about all indexers does not change when you enable clustering. But, as of Splunk 6.3, the indexer discovery feature allows for forwarders to contact a cluster master and simply ask it "what indexers should I connect to?" Then when you add new indexers to the cluster, the forwarders learn of them automatically.

View solution in original post

dwaddle
SplunkTrust
SplunkTrust

In a distributed, non-clustered, environment the answer is a resounding 'no'. The various indexers have no knowledge of each other, there is no shared state across indexers. Both the search heads and the forwarders must be given knowledge of all the indexers. If you add an indexer and only configure it into your search head for distributed search, then it will get no data at all. You can't feed it data without making changes to your forwarders to send data to it.

When you enable clustering, this gives the indexer peers knowledge of each other, but only for the purpose of making redundant copies. An indexer can make an additional copy of data at a peer, but it cannot "migrate" its data to that peer. Come search time, an indexer bucket has but one primary copy, and it is only the primary copy that is searched. Any additional secondary copies do not participate in the search.

For the most part, the requirement that the forwarders know about all indexers does not change when you enable clustering. But, as of Splunk 6.3, the indexer discovery feature allows for forwarders to contact a cluster master and simply ask it "what indexers should I connect to?" Then when you add new indexers to the cluster, the forwarders learn of them automatically.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...