Deployment Architecture

After upgrading my Indexer Cluster to 6.5.1, why is the replication status "pending"?

rafamss
Contributor

Hi dears,

I have * 21 indexers * in my Splunk environment running in index cluster mode. After upgrading the whole site from version 6.3.1 to version 6.5.1, I have the problem with ** replication data **. Invariably some machines fall Down and stay with Pending Status, and then these machines come back Up with Status Searchable. This process, so to speak, occurs several times.

Could someone tell me what that could be? I put two attachments to help.

dxu_splunk
Splunk Employee
Splunk Employee

You'll need to keep indexers of the same cluster within one minor version of each other.

So, 6.4.X indexers will be okay with 6.5.x indexers, however 6.3.x and 6.5.x indexers will not be guaranteed (and indeed 6.3.x <-> 6.5.x+ replication is intentionally broken)

jcrabb_splunk
Splunk Employee
Splunk Employee

You may want to consider going to 6.5.2 as there are two bugs that can impact a busy environment.

6.5.2 Release Notes

  • SPL-134427, SPL-133450: 6.5+ splunk does full bundle replication everytime - slowing down the system
  • SPL-131398, SPL-132804, SPL-132805, SPL-132807, SPL-132890: Search head cluster contention on Linux due to poor hashing inside OpenSSL's error container.

It is possible that either of these are causing some contention which results in a peer timing out while trying to communicate with the Cluster Master. The result of that scenario is a peer with a "Status" that is fluctuating.

If that doesn't correct the issue and you have more than 20-30K buckets per indexer, some timings may need to be adjusted but I would highly encourage you to upgrade first.

Jacob
Sr. Technical Support Engineer
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...