Splunk Search

How do we migrate from single instance to an index and search head cluster

ritsma
Engager

We are trying to move from a single instance of splunk to a clustered environment. We created the cluster as per the documentation only to find out we also need a search head cluster which isn't created when you create a cluster of search heads. So now we are wondering what other items that we are missing. Is there a complete step by step doc on how to do this?

ie in the end we want a clustered environment that looks the same as our single instance.
questions we have so far
1. How do users, dashboards and alerts get migrated over to the cluster, do they go on all systems, or just search heads, or just clustered search heads or a search head cluster?
2. Once we have created a cluster of search heads we need to create a search head cluster, can we do this through the gui or is it all cli?
3. do we need a deployment server to create user dashboards and add users, or do we only need one once we are ready to deploy apps?
4. should the search head cluster even be part of the indexer cluster, or is this handled outside of that?

Any help would be appreciated.

Moving from single instance of splunk spl01 system to clustered indexers and clustered search heads, ie spl01 and spl02 with master of spl00, and search heads of spl11 and spl12

Tags (1)
0 Karma

Raghav2384
Motivator

Hello @ritsma,

1.lets talk about SHC, when you build a search head cluster, rules is minimum of 3 search heads + deployer. All the content you want to migrate from standalone to SHC, put them on deployer (except search app) and push to your search head cluster members.
Example: put the apps other than search app on deployer @ $Splunk_home/etc/shcluster/apps/

And etc/users under $splunk_home/etc/shcluster/users etc. You maintain content on deployer that needs to be pushed to SHC. As far as search app, move the content to a new app and make the artifacts global so that It's available on search app on your SHC.

2.Read the docs on how to migrate to SHC and how to add/initialize new or existing Splunk instances as Search head cluster members.
3.you use deployer as the master node for SHC
4.indexer cluster is managed by cluster master and SHC is managed by Deployer. They work hand in hand from a cluster functionality perspective.

If you look at the SHC members server.conf, you will see two stanzas, one for SHC and one for indexer clustering. In the indexer clustering stanza, all your search heads are listed as cluster members with mode = search head

5.now migrating standalone indexed data to indexer cluster is slightly different. You push you r indexes.conf from cluster master to all peer nodes but data has to be distributed to your indexers in cluster manually. So if you have 10 buckets on standalone and if you want to migrate them to a indexer cluster with 5 peers, you will copy 2 buckets to each peer. NOTE, bump the bucket id's before migration to avoid conflicts and also roll the hot buckets to warm before you migrate.

Docs has this I formation covered in detail.

http://docs.splunk.com/Documentation/Splunk/6.4.2/DistSearch/SHCarchitecture
http://docs.splunk.com/Documentation/Splunk/6.4.2/DistSearch/SHCconfigurationoverview
http://docs.splunk.com/Documentation/Splunk/6.4.3/Indexer/Migratenon-clusteredindexerstoaclustereden...
http://docs.splunk.com/Documentation/Splunk/6.4.2/Indexer/Clusterdeploymentoverview

Hope this helps!

Thanks,
Raghav

0 Karma

ritsma
Engager

Hello Raghav,

Thanks for the reply, just to confirm and clear up a point or two:
To create our env we need the following functions:

1 license master server
1 SHC deployment server
3 search head servers
1 index cluster master
1 Distributed Management Console server
2 indexers
1 forwarder deployment server

Am I missing anything?

If no, can we put the following functions on 1 system (spl00)?
1 license master server
1 SHC deployment server
1 index cluster master
1 Distributed Management Console server

We then need to create 2 indexers and add them to the index cluster (spl01 and spl02)
We then need 3 search heads (spl11, spl12 and spl13)

Is this step necessary, required or should not be done?
We then add the 3 search heads to the index cluster and they become distributed search heads for the index cluster.

We then change the distributed search heads into SHC by entering the commands schluster-config and schluster-captain

At this point we would then migrate the existing dashboards, roles, users and alerts to one of the SHC members, or to
the SHC deployer? If the deployer, then how would users create their own dashboards? We currently have no apps other than
the defaults.

Thanks

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...