Deployment Architecture

Restarting indexer cluster peers

nawazns5038
Builder

I have tried the rolling restart of the cluster peers and it doesn't solve the problem I'm facing and the manual restart of one of the cluster peers gave the expected result.

  1. Can we restart a single search peer by keeping the cluster in maintenance mode ?
  2. Can we restart a single search peer by keeping the cluster in maintenance mode and keeping that particular peer offline ?

Problem I'm facing :
change of ulimits .... ulimit -n 65536, I have used this command and the change isn't happening unless Splunk is restarted. If the cluster is restarted by rolling restart I cannot see the change in the ulimtis

Please help !!
Thanks

0 Karma

gjanders
SplunkTrust
SplunkTrust

I have used this command and the
change isn't happening unless Splunk
is restarted. If the cluster is
restarted by rolling restart I cannot
see the change in the ulimtis

That would be expected behaviour, for a new ulimit to be respected you would need to create a process from scratch from a newly logged in shell session (or reboot the server).
If Splunk triggers the restart it will have to fork the existing process which has the old ulimit to trigger the restart...(and therefore the restarted process also has the old ulimit if you restart through Splunk)

Once-off you are going to need to restart the indexer cluster peers from the CLI, you could just run splunk offline and then bring that peer back online once done.
Obviously you need to do this during a maintenance window of some kind.

0 Karma

nawazns5038
Builder

Hi @garethatiag ,

You mean to say bring a peer offline and make changes and reboot the instance ? and bring back online.
Won' t that effect the cluster or is it a safe way to reboot the cluster

0 Karma

gjanders
SplunkTrust
SplunkTrust

It will effect the cluster, the offline command will advise the master that the peer is going offline and therefore make the appropriate arrangements to ensure the cluster remains searchable while the peer is offline.

It will also briefly apply maintenance mode (for a period of time, see the linked documentation for more information).

This will be safer than splunk stop, splunk start on a cluster member.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...