What is the best sequence for a Splunk distributed deployment shutdown?

[Gursimar_singh]

We have a distributed deployment consisting of  2 Search heads, 1 indexer, Deployment server, 2 Heavy Forwarders, Universal Forwarders and a Syslog server. We need to shut it down and then boot it back up. What is the best sequence to shutdown and boot up the environment gracefully? 

Also anything to keep in mind while doing so to avoid errors. 

[PickleRick]

You can shut down the servers in virtually any order. Just be aware that the functionality of the downed component will not be available. But since you want to shut the whole environment down, you probably don't mind that.

[isoutamo]

Hi

It's just like @PickleRick said. One comment to that. When you have shutdown indexer you cannot ingest any new events. For that reason I prefer to start from out circle like UFs then HFs, then other splunk infra nodes and indexer as a last one. Then you will have as much events on it as possible (e.g. for further debug purpose). And when you will start the whole environment I use the reverse order for the same reason.

If you just want to restart then any order is a good order.

BUT if you are doing "live update" (cannot do it really as you have only one indexer), you must follow up the correct order. You can found it from here or from Splunk Lantern.

r. Ismo

[PickleRick]

True. On the other hand, if you have some "transient" sources, like syslog, the longer your forwarders are down, the more events you can't receive and queue so it's up to the particular architecture. Technically nothing should "break" just because you shut down indexers before search-head or vice-versa.

Anyway, if the downtime is planned for splunk upgrade, it can be performed one node at a time, not necessarily needing to shut down the whole setup.  (of course the proper order should be maintained).