Getting Data In

Why is our one indexers in a multisite cluster consuming more disk space than the other indexers?

anandhscareer
New Member

In our environment, we have one master node and four indexers, out of which 3 indexers are located in a production site and the last indexer is in a Disaster Recovery site. The setup is made in such a way that all the production indexers data (three indexers data) are getting replicated to only one indexer which is in DR site.

We have allocated 400 GB for each indexer, but for the last indexer which is located in DR region, it is consuming more disk space and sometimes it's going down itself. Out of 400 GB, it's consuming nearly 395 GB and hence it results in the indexer to go down due to the search and replication factor not being met on the master node. We are also getting the error message below from the other indexer as "search peer failed to make bucket".

So kindly let me know how to fix the issue. I have attached screenshot for your reference.alt text

0 Karma

sowings
Splunk Employee
Splunk Employee

As data arrives in each of your indexers in the primary site, the data is arranged into "buckets". In order to satisfy the DR requirements of your configuration, these buckets have to be replicated over to your DR site. Note now that you have three indexers feeding one. All other things being equal, you're asking for one host with 400GB of space to support the disk consumption needs of three other hosts also with 400GB each. The DR host doesn't have nearly the amount of disk space to keep up.

Your choices are:

  • Lower the retention settings (i.e. how long to keep data) so that you don't flood the DR indexer.
  • Give the DR indexer more space.
  • Create more DR indexers.
0 Karma

jkat54
SplunkTrust
SplunkTrust

What is your site_replication_factor?

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Is the DR indexer three times the size of the Production indexers? If not, that may be your problem.

---
If this reply helps you, Karma would be appreciated.
0 Karma

jkat54
SplunkTrust
SplunkTrust

Yeah that's it because they said its 400GB on each indexer. Hence why I asked what the site replication factor is. @amandhscareer, lets say your 3 indexers each have a unique copy of 1 bucket. Your DR indexer will get a copy of each bucket, making 3 buckets total.

See this document as there is also a site_search_factor you should consider as well.

http://docs.splunk.com/Documentation/Splunk/6.2.1/Indexer/Multisitearchitecture

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...