Deployment Architecture

Restarting indexer cluster peers

nawazns5038
Builder

I have tried the rolling restart of the cluster peers and it doesn't solve the problem I'm facing and the manual restart of one of the cluster peers gave the expected result.

  1. Can we restart a single search peer by keeping the cluster in maintenance mode ?
  2. Can we restart a single search peer by keeping the cluster in maintenance mode and keeping that particular peer offline ?

Problem I'm facing :
change of ulimits .... ulimit -n 65536, I have used this command and the change isn't happening unless Splunk is restarted. If the cluster is restarted by rolling restart I cannot see the change in the ulimtis

Please help !!
Thanks

0 Karma

gjanders
SplunkTrust
SplunkTrust

I have used this command and the
change isn't happening unless Splunk
is restarted. If the cluster is
restarted by rolling restart I cannot
see the change in the ulimtis

That would be expected behaviour, for a new ulimit to be respected you would need to create a process from scratch from a newly logged in shell session (or reboot the server).
If Splunk triggers the restart it will have to fork the existing process which has the old ulimit to trigger the restart...(and therefore the restarted process also has the old ulimit if you restart through Splunk)

Once-off you are going to need to restart the indexer cluster peers from the CLI, you could just run splunk offline and then bring that peer back online once done.
Obviously you need to do this during a maintenance window of some kind.

0 Karma

nawazns5038
Builder

Hi @garethatiag ,

You mean to say bring a peer offline and make changes and reboot the instance ? and bring back online.
Won' t that effect the cluster or is it a safe way to reboot the cluster

0 Karma

gjanders
SplunkTrust
SplunkTrust

It will effect the cluster, the offline command will advise the master that the peer is going offline and therefore make the appropriate arrangements to ensure the cluster remains searchable while the peer is offline.

It will also briefly apply maintenance mode (for a period of time, see the linked documentation for more information).

This will be safer than splunk stop, splunk start on a cluster member.

0 Karma
Get Updates on the Splunk Community!

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...