Cluster Master

[garima_chauhan]

Hi,

I am using cluster set up with all instances of Splunk v5.0.5. I wanted to know what happens if the master goes down. The following link explains that the peers and search head will continue to work without any problem. A problem would only occur if one of the peers also goes down, in which case a standby master node can be of help.

http://docs.splunk.com/Documentation/Splunk/5.0.5/Indexer/Whathappenswhenamasternodegoesdown

I have a few questions, in the event that master node goes down:

1. Is there any explicit time for which the peers and search head continue to work properly if all these are up and working.
2. What else, apart from one of the peers going down, would arouse the need of a stand by cluster master?

Please confirm if my understanding is correct. Also, do answer my queries. Any help would be appreciated.

[joebensimo · Accepted solution]

In my experience, with Splunk v5.0.x, there is no explicit time for which the search head and peers continue to work after a master goes down. I've had them continue to work, but also had them fail in under a half hour. As the document says, "treat a downed master as a serious failure" and start fixing the problem immediately.

Also, with Splunk v5.0.x, I've found that bringing the master back online may put the cluster in a bad state for a while.

The most common problems I've seen when a master is down is that searches return incomplete data; however, I've also see a cluster become completely unsearchable.

In my experience, v5 handles clustering very poorly. I highly recommend you upgrade to v6.