Getting Data In

Why is our one indexers in a multisite cluster consuming more disk space than the other indexers?

New Member

In our environment, we have one master node and four indexers, out of which 3 indexers are located in a production site and the last indexer is in a Disaster Recovery site. The setup is made in such a way that all the production indexers data (three indexers data) are getting replicated to only one indexer which is in DR site.

We have allocated 400 GB for each indexer, but for the last indexer which is located in DR region, it is consuming more disk space and sometimes it's going down itself. Out of 400 GB, it's consuming nearly 395 GB and hence it results in the indexer to go down due to the search and replication factor not being met on the master node. We are also getting the error message below from the other indexer as "search peer failed to make bucket".

So kindly let me know how to fix the issue. I have attached screenshot for your reference.alt text

0 Karma

Splunk Employee
Splunk Employee

As data arrives in each of your indexers in the primary site, the data is arranged into "buckets". In order to satisfy the DR requirements of your configuration, these buckets have to be replicated over to your DR site. Note now that you have three indexers feeding one. All other things being equal, you're asking for one host with 400GB of space to support the disk consumption needs of three other hosts also with 400GB each. The DR host doesn't have nearly the amount of disk space to keep up.

Your choices are:

  • Lower the retention settings (i.e. how long to keep data) so that you don't flood the DR indexer.
  • Give the DR indexer more space.
  • Create more DR indexers.
0 Karma

SplunkTrust
SplunkTrust

What is your site_replication_factor?

0 Karma

SplunkTrust
SplunkTrust

Is the DR indexer three times the size of the Production indexers? If not, that may be your problem.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

SplunkTrust
SplunkTrust

Yeah that's it because they said its 400GB on each indexer. Hence why I asked what the site replication factor is. @amandhscareer, lets say your 3 indexers each have a unique copy of 1 bucket. Your DR indexer will get a copy of each bucket, making 3 buckets total.

See this document as there is also a site_search_factor you should consider as well.

http://docs.splunk.com/Documentation/Splunk/6.2.1/Indexer/Multisitearchitecture

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!