topic Migration instructions from single install to distributed? in Deployment Architecture

Migration instructions from single install to distributed?

robertlynch2020 — Mon, 10 Jun 2019 15:27:36 GMT

Hi
I have a single install (Everything on one machine).

I want to go to one search head and 2 indexers (non clustered) multiple machines.

Is there a set of instruction on how to do this, the doc is great but there seem to be so many options that I get lost.
I am looking for step 1 2, 3..etc...

I also have a question like:
If I change the current install from SH+Indexer -> indexer and create a separate search head(I think this is the best way),
do I have to reinstall all my apps onto the new search head?

Regards
Robert

Re: Migration instructions from single install to distributed?

jnudell_2 — Mon, 10 Jun 2019 19:23:52 GMT

Hi Robert,

There are plenty of Splunk documents on how to setup a distributed configuration (which you've probably encountered already):
New install: https://docs.splunk.com/Documentation/Splunk/7.2.6/DistSearch/Overviewofconfiguration

However, Google isn't really providing insight when migrating from a standalone to a distributed environment.

I think the easiest path would be as follows:
1. Install new Splunk instance (this will be the search head)
Linux: https://docs.splunk.com/Documentation/Splunk/7.3.0/Installation/InstallonLinux
Windows: https://docs.splunk.com/Documentation/Splunk/7.3.0/Installation/ChoosetheuserSplunkshouldrunas
2. Configure it to send data to the old instance/indexer-to-be
https://docs.splunk.com/Documentation/Splunk/7.2.6/DistSearch/Forwardsearchheaddata
3. Configure it to use the old instance/indexer-to-be as a search-peer (same thing as indexer, different terminology)
https://docs.splunk.com/Documentation/Splunk/7.2.6/DistSearch/Configuredistributedsearch
4. Copy your apps from the old instance to the new search head (/opt/splunk/etc/apps)
App migration reference: https://docs.splunk.com/Documentation/Splunk/latest/DistSearch/Migratefromstandalonesearchheads
6. Restart your search head
7. Review .conf files in /opt/splunk/etc/system/local to determine what needs to be moved to the search head

That's a rough overview, but should get you to where you want to be.

Re: Migration instructions from single install to distributed?

richgalloway — Mon, 10 Jun 2019 19:33:00 GMT

If it was as easy as 1,2,3 the docs would say so. The docs have a lot of options because there are a lot of variables.

If I was in your shoes, I'd make the existing server an indexer and add new servers to act as search head and second indexer. Usually, it's not necessary to re-install apps - just transfer $SPLUNK_HOME/etc/apps from the old server to the new one. There are caveats so read the docs.
The second indexer will start out empty, but will accumulate data over time. Until then, however, searches won't benefit from the second indexer. Better is to cluster the indexes and balance the indexes from the start, but that's something for Professional Services to handle for you.

Re: Migration instructions from single install to distributed?

robertlynch2020 — Wed, 12 Jun 2019 16:03:23 GMT

Thanks for this, i will try it and get back

Re: Migration instructions from single install to distributed?

robertlynch2020 — Wed, 09 Oct 2019 13:44:42 GMT

Thanks for this answer.
We have done the following. One new search head and 2 indexers. The 1st Indexer is the old production, but when we start it up all the datamodels start to rebuild. Is there a way to get the datamodels not to rebuild?

Thanks
Robert

Re: Migration instructions from single install to distributed?

robertlynch2020 — Wed, 09 Oct 2019 14:00:23 GMT

Thanks
Robert

Re: Migration instructions from single install to distributed?

woodcock — Wed, 09 Oct 2019 14:00:23 GMT

Do not use Windows for your infrastructure here.

Re: Migration instructions from single install to distributed?

robertlynch2020 — Fri, 11 Oct 2019 11:13:28 GMT

noted we are on UNIX for everything

Re: Migration instructions from single install to distributed?

woodcock — Fri, 11 Oct 2019 12:56:31 GMT

You have saved yourself much avoidable pain.

Re: Migration instructions from single install to distributed?

woodcock — Fri, 11 Oct 2019 12:57:29 GMT

If you are going to use them, they must be rebuilt.

Re: Migration instructions from single install to distributed?

robertlynch2020 — Fri, 11 Oct 2019 14:04:03 GMT

Ok thanks - that is disappointing to here, it looks like the migration process will take ~48 hour of MAX CPU on a 60 core machine.

We have ~20 Data models.

Lucky we are on a very very good machine, otherwise we would have to stop production or do a parallel run