Deployment Architecture

Turn-off Muster Node/License Master/Deployer Machine

LM_ACN
Engager

Hello everyone,

i'm actually dealing with an infrastructure composed by three search heads, two indexer and a single instance with master node, license master and deployer together.

Now, for maintainance activity, the master machine will be turned off for 48 hours.

I have few questions about the possible consequences and preliminary step:

  1. It is sufficient to stop the splunk service on the istance?
  2. Does indexing still continue?
  3. If indexing stop, which other file to put inside a back up splunk machine in addition to server.conf and master-app, considering also deployer and license master capabilities?
  4. If everything goes right, it is sufficient to start the splunk service on the master machine (primary)?

Thanks in advance.

Tags (1)
0 Karma
1 Solution

moliminous
Path Finder

Firstly, I'd like to point out it's recommended to have the SHC Deployer be on it's own.

Secondly, I will assume you have a Search Head Cluster and Index Cluster since you mentioned master node and deployer. You will have 72 hours until search is disabled due to the License Master not being available. If you expect the LM will be down longer than your 48 hours, I would first migrate the License Master role to a new standalone (or at least another machine) if possible.
https://docs.splunk.com/Documentation/Splunk/7.3.2/Admin/Aboutlicenseviolations

  1. Yes, it is sufficient to stop the Splunk service on the instance. The cluster will be at risk if you lose a search peer (Indexer) however everything will continue to function without the Cluster Master, SHC Deployer, and the License Master (up to 72 hours).

  2. Yes, indexing will continue as long as the replication and search factors are being met, even after missing the License Master for more than 72 hours.

  3. Indexing does not stop, so regarding other files to backup, it's a good idea to always backup /opt/splunk/etc/ (recursive) on every Splunk server in your deployment on a regular basis. Regarding specific files, I would make sure you have your .lic license file handy to being able to switch LMs if needed, and the SHC Deployer you would want a backup of everything under /opt/splunk/etc/shcluster/ (recursive).

  4. Not sure what 'everything' entails, but assuming the server is alive and well and effectively the same as it was prior to stopping Splunk (new version of Splunk is fine), then yes you can start Splunk on that machine and be running fine again. The Indexers will continue sending heartbeats and will detect when the Cluster Master is back up. The CM will do its thing, making sure the cluster is healthy and do bucket fixup as needed.
    https://docs.splunk.com/Documentation/Splunk/7.3.2/Indexer/Whathappenswhenamasternodegoesdown

Good luck!

View solution in original post

0 Karma

moliminous
Path Finder

Firstly, I'd like to point out it's recommended to have the SHC Deployer be on it's own.

Secondly, I will assume you have a Search Head Cluster and Index Cluster since you mentioned master node and deployer. You will have 72 hours until search is disabled due to the License Master not being available. If you expect the LM will be down longer than your 48 hours, I would first migrate the License Master role to a new standalone (or at least another machine) if possible.
https://docs.splunk.com/Documentation/Splunk/7.3.2/Admin/Aboutlicenseviolations

  1. Yes, it is sufficient to stop the Splunk service on the instance. The cluster will be at risk if you lose a search peer (Indexer) however everything will continue to function without the Cluster Master, SHC Deployer, and the License Master (up to 72 hours).

  2. Yes, indexing will continue as long as the replication and search factors are being met, even after missing the License Master for more than 72 hours.

  3. Indexing does not stop, so regarding other files to backup, it's a good idea to always backup /opt/splunk/etc/ (recursive) on every Splunk server in your deployment on a regular basis. Regarding specific files, I would make sure you have your .lic license file handy to being able to switch LMs if needed, and the SHC Deployer you would want a backup of everything under /opt/splunk/etc/shcluster/ (recursive).

  4. Not sure what 'everything' entails, but assuming the server is alive and well and effectively the same as it was prior to stopping Splunk (new version of Splunk is fine), then yes you can start Splunk on that machine and be running fine again. The Indexers will continue sending heartbeats and will detect when the Cluster Master is back up. The CM will do its thing, making sure the cluster is healthy and do bucket fixup as needed.
    https://docs.splunk.com/Documentation/Splunk/7.3.2/Indexer/Whathappenswhenamasternodegoesdown

Good luck!

0 Karma

LM_ACN
Engager

Really exhaustive answer!

Thank you 🙂

Luca

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...