Splunk Enterprise Operating System Upgrade Approach

[shocko]

We are running Splunk Enterprise 10.2.3 on prem on Windows 2016 as follows

System	OS	Roles	Virtual	Comments
ServerA	Windows 2016 Standard	Search Head License Master Cluster Master Deployment Server	Yes
ServerB	Windows 2016 Standard	Search Head KV Store	Yes
IndexerA	Windows 2016 Standard	Indexer	No	Local Storage for indexes
IndexerB	Windows 2016 Standard	Indexer	No	Local Storage for indexes

The factors are as follows:

Reaplication=2

Search=2

We need to move everything to Windows 2022 or higher. I also wish to avoid  buying  any new physical servers for the indexers. I'm looking for a steer on any caveats or other approaches I coudl take here from somone who's been through this process or something similar. 

My thinking is as follows

Break down the migration/upgrade as follows

• Handles Indexers first
• Move on to other roles

Approach for the indexers

Option 1

1. Point all forwarders at a single indexer
2. Backup config of indexer targeted for upgrade
3. Do an in-place windows upgrade of one indexer to Windows 2022. Indexes are on dedciated non-os drive and hence should not be impacted by the OS in-place upgrade
4. If working/stable point all forwarders at this indexer and rinse an repeat for the other one

 Option 2

1. Point all forwarders at a single indexer
2. Backup config of indexer targeted for upgrade
3. Rebuild this indexer with Windows 2022. Indexes are on a separate non-os drive so should be unaffected by upgrade
4. Install Splunk Enterprise
5. Restore indexer config
6. If working/stable point all forwarders at this indexer and rinse and repeat for the other one

If all working move on to ServerA which is Search head and KV store

Approach for other roles

1. Snapshot VM offline
2. Run in-place windows upgrade
3. Validate configuration

If all working move onto cluster master

No idea! In-place upgrade seems more risky for this one

Note: we have file-based backups of all configurations and indexes

[shocko]

Thanks for taking the time to respond @PickleRick . I should have indlcuded this and as such have updated the original post:

Replication Factor= 2
Search Factor = 1

We also  have a single site and two indexers (I have corrected the original table to reflect this).

Note: Downtime is not an issue per se as we will do this out of hours

[PickleRick]

In this case you can just take down servers one by one generally in any order.

Just remember to enable maintenance mode on the cluster so it doesn't start rebuilding searchable copies immediately.

[PickleRick]

There are some unknowns here. Do you actually have a 1+1 cluster? What is your RF/SF? What is your data flow? I'm assuming you want to have as little downtime as possible, right?

Remember that forwarders can and will pause inputs if they cannot send to outputs (if those are the "stoppable" inputs; you can't "hang" syslog packets on a wire). Also you will undoubtedly have downtime during SH upgrade.

Generally in a normal scenario you should not be needing to "point" anything anywhere becaue in a well-engineered setup you should be load-balancing outputs over your available indexers so any single-node downtime should not affect ingest flow.

Also, be very careful about the in-place windows upgrades. They can go wrong. I would consider doing backup/restore over a clean server. As long as the names and IPs stay the same, the process should be trivial. Especially if you indeed have index storage on a separate storage which can be easily "plugged into" a new/restored instance.