topic Re: Multi-site Indexer rolling restart - indexer fails to restart/timeout in Deployment Architecture

Multi-site Indexer rolling restart - indexer fails to restart/timeout

DEAD_BEEF — Mon, 03 Feb 2020 01:34:38 GMT

Using Splunk 7.3.3, after I initiated a rolling restart from the cluster master (multi-site indexer cluster), the first indexer began to restart. Then it showed batch adding, then the Indexer Clustering: Master Node page, showed that the indexer failed to restart

[Mon Feb  2 12:47:52 2020] Failed to restart peer=<GUID> peer_name=<hostname>. Moving to failed peer group and continuing.
[Mon Feb  2 12:47:52 2020] Failing peer=<GUID> peer_name=<hostname> timed out while trying to restart.

I did a ping from the CM to this indexer and it returned fine. Connectivity was not an issue before the rolling restart and network connectivity appears to be working fine.

Is there a timeout window or setting I can adjust to better accommodate network latency and give the CM more time to reach the peer?
What does this mean for my rolling restart, will remaining peers be restarted but I should restart this one manually?
How can I list this "failed peer group" to see all systems that may fail to restart?

Re: Multi-site Indexer rolling restart - indexer fails to restart/timeout

maraman_splunk — Mon, 03 Feb 2020 11:02:09 GMT

Hi, you are probably looking at the restart timeout setting on the CM (see link text and link text)

[clustering]
restart_timeout = time_in_sec 
# default is 60, probably a good idea to really increase here (to avoid the cluster to go in fix mode)  but still adapt it to the time it usually take for a idx to restart (use something like 3600 if you really want not to restart in that case but obviously if your idx crash in the middle of the restart, this will take more time to detect)

Re: Multi-site Indexer rolling restart - indexer fails to restart/timeout

hmallett — Wed, 30 Sep 2020 03:59:31 GMT

From the error, if the indexer did restart without manual intervention, I would guess that the restart of the indexer took longer than the restart_timeout defined in the cluster master's server.conf. By default this is set to 60 seconds, and I have seen indexers take much longer than this to restart.

Can you see from splunkd.log on the indexer how long the restart actually took? If it's longer than 60 seconds, then you might want to extend your restart_timeout (https://docs.splunk.com/Documentation/Splunk/7.3.3/Indexer/Userollingrestart#Handle_slow_restarts)

Re: Multi-site Indexer rolling restart - indexer fails to restart/timeout

DEAD_BEEF — Wed, 05 Feb 2020 08:52:50 GMT

Most indexers were taking 15-20 mins. I will try adjusting the restart_timeout value but this is the first time I've seen these errors and I have restarted this cluster many times with each taking 15-20 mins just like always. That's what prompted me to ask about this issue.

So this setting needs to be changed on the CM's server.conf, not the indexers themselves?

Re: Multi-site Indexer rolling restart - indexer fails to restart/timeout

DEAD_BEEF — Wed, 05 Feb 2020 08:53:37 GMT

I will try adjusting this. Each idx takes on average 15-20 mins, my current timeout setting is 15mins, so maybe I just expand it to 30m to be safe?

Re: Multi-site Indexer rolling restart - indexer fails to restart/timeout

maraman_splunk — Wed, 05 Feb 2020 09:23:12 GMT

Yes, it is a CM setting. 30 min(1800s) seem to be appropriate for your env.

Re: Multi-site Indexer rolling restart - indexer fails to restart/timeout

DEAD_BEEF — Sun, 09 Feb 2020 04:28:32 GMT

Just finished a rolling restart and no errors anymore after increasing the timeout to 30mins. Thank you both for the assistance!