Guidance on Migrating Splunk Enterprise Search Head to Azure Cloud

[KingUs80]

Hello,

We are in the process of fully migrating our Splunk Enterprise deployment to the Azure Cloud and will no longer be using Splunk Enterprise on-premises. Specifically, I have a question about moving the search head and all its associated components to the cloud without causing disruptions.

While we found a Work Instruction on the Splunk website, it wasn't clear enough to follow, and we're concerned about minimizing downtime during the migration process.

Could anyone provide guidance (step-by-step guidance) or best practices for migrating a Splunk search head and its components to the Azure Cloud, ensuring no service interruptions during the transition?

Your help would be greatly appreciated!

[PickleRick]

1. If you're migrating into another environment there will be issues. You can't completely seamlessly move from one point to another without anyone noticing. Even if you have some form of HTTP LB in front of your SH(S) so that you can simply point it to other backend, you are bound to at least break existing browsing sessions, there will be issues with replicating last minute changes between those environment and so on.

2. There is no way to give a precise step by step fool-proof instructions which can be just executed without knowing what you're doing.

3. You're talking about migrating just a search head but you're posting in the Splunk Cloud section of the forum so it's not clear what you actually wanna do.

Overall, as usual with more complicated stuff and people who ask question which seem to be significantly above their knowledge/expertise level (I'm not trying to offend you here - I'm trying to save you time/money by preventing you from breaking your stuff) - I'd advise to seek help from either Professional Services or your local friendly Splunk Partner who has certified and experienced engineers who will help you get through this process.

[KingUs80]

@PickleRick  Thank you for the input!

[mattymo]

Hello!

Is it a single search head or search head cluster?

Have your indexers already been migrated to Azure? Will the SH(C) be searching Azure or On-Prem indexers as well?

What "components" do you rely on most on this SH(C)? Premium apps like ES or ITSI? or just Splunk Enterprise apps?

I would probably:

- back up the apps and kvstore if needed
- build the new SH/SHC in the cloud
- restore configs
- cut over DNS or point users in a uniform fashion to the new SH during a Maintenance Window
- shut down the old SH. 

Curious what docs you ended up in, always worth feedback at the bottom of the docs page if the topic or guidance needed was missing. 

[KingUs80]

@mattymo Do the answers to your questions affect the logic of your previous suggestions? If possible, could you please clarify?

I would probably:

- back up the apps and kvstore if needed
- build the new SH/SHC in the cloud
- restore configs
- cut over DNS or point users in a uniform fashion to the new SH during a Maintenance Window
- shut down the old SH. 

[PickleRick]

Your answers are againt a bit confusing.

You say that "indexers have been migrated yet". Does it mean they have been already or they haven't been yet?

Anyway, your general plan looks pretty decent. It's always the details, like making sure you have proper network connectivity - SH->idx, SH->LM, probably also MC->SH. If you install new machines with new names and new IPs and you have either IP-based access rules or allowed SANs, you might have problems. I've even seen situations when TLS connections wouldn't be allowed because a perimeter IPS was forbidding connection with local CA-issued certs.

So prepare to at least do a trial launch for test users to verify if everything's working properly before going prod.

[KingUs80]

Hello,

Thank you @mattymo for taking the time to provide input on my question. These are the answers I received from the Splunk team after discussing your questions with them.

Is it a single search head or search head cluster?

- No cluster all single search heads

Have the indexers already been migrated to Azure?

- No indexers have been migrated yet

Will the SH(C) be searching Azure or On-Prem Indexers as well?

- SH will be searching Azure indexers

What "components" do you rely on most on this SH(C)? Premium apps like ES or ITSI? or just Splunk Enterprise apps?

- SH will rely on both premium apps and Splunk Enterprise apps

[mattymo]

Is it a single search head or search head cluster?

- No cluster all single search heads

Then really it should just be a user migration to the new SH that is running side by side from a backup as descibed. With them being singletons, theres already a reduced amount of redundancy or expectation of no disruption, so should just be an organized migration and shut down. 

Have the indexers already been migrated to Azure?

- No indexers have been migrated yet

So will the Search Heads migrated be searching back to onprem when cutover is done? if so, as mentioned, you need to have the networking right, so another reason to build the sh in azure along side existing, confirm search and comms work. 

Will the SH(C) be searching Azure or On-Prem Indexers as well?

- SH will be searching Azure indexers

So when you build the Azure SH, they will be searching net new indexers in Azure? Do they need to Search Onprem too? Just gotta nail the config and networking. 

What "components" do you rely on most on this SH(C)? Premium apps like ES or ITSI? or just Splunk Enterprise apps?

- SH will rely on both premium apps and Splunk Enterprise apps

Well, ES and ITSI are their own beasts, see documentation for those. Enterprise apps will depend on what needs to persist and be migrated. 

Definitely ideal to involve experienced services folks or experienced Splunk Admins.

Either way it is basically a build new SH, copy configs over, validate, migrate users over. All the traps you may encounter along the way should mostly be resolved in the standing up of the new SH and then getting configs running along side your existing.  This is nuanced and is why you wont really find a 1:1 migration guide, cause with Splunk, "it depends". 

The amount of disruption will be mitigated by having good understanding of the major workloads running in the environments (montioring console can help with by app breakdowns) and what needs to be carried over to the new enviro and what can/needs to be cleaned up to reduce work needed in migration.