Deployment Architecture

What is the best practice for bringing down a search head cluster member?

myandow
Path Finder

I'm trying to find some guidance on the best way to bring down a member of a search head cluster without impacting any of the searches that are currently running on it. What I have found in my limited testing is that following this documentation: http://docs.splunk.com/Documentation/Splunk/6.2.2/DistSearch/Removeaclustermember will cause the searches running on the search head to be orphaned and they are no longer viewable in the job manager on the remaining active cluster members.

Is there a best practice when preparing for a scheduled maintenance to bring down a member of a search head cluster so that there will be no impact to the system as a whole?

1 Solution

sloshburch
Splunk Employee
Splunk Employee

Alternate solution based on the discussion we had (finally documenting it):

When doing maintenance, configure the load balancer to stop sending users to the search head that is about to be worked on. That way no new ad hoc searches will be generated that anymore. After about ten minutes (the default ttyl of search artifacts) go ahead and do the maintenance on that host. Then enable the load balancer to send traffic there again. Rinse and repeat with your next shc member.

The shc itself will ensure scheduled searches complete successfully - so you don't have to worry about losing a scheduled search. The load balancer will ensure no ad-hoc searches land on that host. The shc will coordinate reelection of captain as needed.

View solution in original post

0 Karma

sloshburch
Splunk Employee
Splunk Employee

Alternate solution based on the discussion we had (finally documenting it):

When doing maintenance, configure the load balancer to stop sending users to the search head that is about to be worked on. That way no new ad hoc searches will be generated that anymore. After about ten minutes (the default ttyl of search artifacts) go ahead and do the maintenance on that host. Then enable the load balancer to send traffic there again. Rinse and repeat with your next shc member.

The shc itself will ensure scheduled searches complete successfully - so you don't have to worry about losing a scheduled search. The load balancer will ensure no ad-hoc searches land on that host. The shc will coordinate reelection of captain as needed.

0 Karma

myandow
Path Finder

I just wanted to add that in order to avoid impacting any long running adhoc searches you should first check the search activity in the DMC for the search head in question and verify there are no active adhoc searches. After that has been verified then wait 10 minutes before starting the maintenance.

0 Karma

sloshburch
Splunk Employee
Splunk Employee

It sounds like no one knows, and as such, there might not be a good way to do it. I have heard that this is such a valid question that it's recommended you generate a P4 ticket for such a feature to be added as a Core Improvement Request.

0 Karma

myandow
Path Finder

As a follow up question, will a rolling restart of the cluster kill off running searches as well?

0 Karma

sloshburch
Splunk Employee
Splunk Employee

It's not documented so let's assume yes (I haven't tested it so I'm not 100% confident).

This is a good opportunity to post on the bottom of the corresponding documentation page to ask this behavior to be articulated.

As such, I've posted your question and a link to this page on the respective page: http://docs.splunk.com/Documentation/Splunk/6.4.2/DistSearch/RestartSHC

The docs team is amazing and I have full confidence they'll be able to help out.

0 Karma

sloshburch
Splunk Employee
Splunk Employee

After hunting down this info even more, some additional details were highlighted:

  • "for scheduled searches, the captain will try to reschedule if it never gets a SUCCESS from the member. Also, if a node is removed it stops heartbeating to the captain and once it reaches heartbeat_timeout (server.conf) captain removes all the searches delegated to that peer as failed searches leading to retry of search delegation" So that means is there's a rolling restart and a scheduled search gets killed, the captain will see it got killed and kick it off again. If it's an ad hoc search, the user would be navigated to the another SHC member where they have to run it again.
  • Also, in terms of ad hoc searches: "It runs on the member on which the user executed it, so if that member goes down, the search is lost. Ad hoc search artifacts only stick around for ten minutes (by default), so, by nature, they are highly ephemeral anyway"

Let us know if there are any other questions!

0 Karma

sloshburch
Splunk Employee
Splunk Employee

It might be worth highlighting that I think the ad hoc behavior is the same if we're talking a stand alone SH, SHP or SHC. If Splunk stops, it will stop the search and users would not be notified said search was terminated.

0 Karma

myandow
Path Finder

We have a lot of users that kick off long running searches and then background them. Are those still considered ad-hoc searches? It is not ideal if a user does this, and then never gets a notification that the search completed or failed to complete, due to a scheduled maintenance.

0 Karma

Steve_G_
Splunk Employee
Splunk Employee

If the user kicks it off, then, yes, it's an ad hoc search.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...