Follow steps below:
Kindly test on dev environment first to check all config and indexed data is available after upgrade of VM.
Take backup of all instances.
You need to upgrade tiers in specific order and within each tier each node should be upgraded at same time:
Follow the order below for upgrades:
1. Master- stop splunk on the master, upgrade the VM and start splunk again.
Check all the cluster status in the Monitoring Console.Check if any errors in internal logs.
2. Search Head -
a.stop splunk on 1 search head, upgrade VM and start splunk again.
Now make that search head as captain and then repeat step a for all other search heads
Enable maintenance node on master.
Stop all the indexers.
Start indexers and disable maintenance-mode.
Let me know if this helps!!
Doing this as a comment, not answer, because this is not really canonical.
Splunk is only very loosely coupled to the OS and upgrades of the OS are not particularly important to Splunk. If there's no clustering in your environment, then you can do whatever, IMO, with the caveat that you probably really want all the OSes to be of nearly the same version. (If for no other reason than management should be easier).
With indexer clusters (and perhaps search head clustering) you'll want those boxes - the CM and indexers, or whatever is involved with SHC, to be upgraded all at once or at least within a relatively short time. Of course, to upgrade an indexer cluster, maintenance mode and all that needs to be done just because the expected downtime will likely be long enough you don't want panic bucket fixings...
Otherwise, it really shouldn't be a big deal.