Migration instructions from single install to distributed?

[robertlynch2020]

Hi
I have a single install (Everything on one machine).

I want to go to one search head and 2 indexers (non clustered) multiple machines.

Is there a set of instruction on how to do this, the doc is great but there seem to be so many options that I get lost.
I am looking for step 1 2, 3..etc...

I also have a question like:
If I change the current install from SH+Indexer -> indexer and create a separate search head(I think this is the best way),
do I have to reinstall all my apps onto the new search head?

Regards
Robert

[richgalloway]

If it was as easy as 1,2,3 the docs would say so. The docs have a lot of options because there are a lot of variables.

If I was in your shoes, I'd make the existing server an indexer and add new servers to act as search head and second indexer. Usually, it's not necessary to re-install apps - just transfer $SPLUNK_HOME/etc/apps from the old server to the new one. There are caveats so read the docs.
The second indexer will start out empty, but will accumulate data over time. Until then, however, searches won't benefit from the second indexer. Better is to cluster the indexes and balance the indexes from the start, but that's something for Professional Services to handle for you.

[robertlynch2020]

Hi

Thanks for this answer.
We have done the following. One new search head and 2 indexers. The 1st Indexer is the old production, but when we start it up all the datamodels start to rebuild. Is there a way to get the datamodels not to rebuild?

Thanks
Robert

[jnudell_2 · Accepted solution]

Hi Robert,

There are plenty of Splunk documents on how to setup a distributed configuration (which you've probably encountered already):
New install: https://docs.splunk.com/Documentation/Splunk/7.2.6/DistSearch/Overviewofconfiguration

However, Google isn't really providing insight when migrating from a standalone to a distributed environment.

I think the easiest path would be as follows:
1. Install new Splunk instance (this will be the search head)
Linux: https://docs.splunk.com/Documentation/Splunk/7.3.0/Installation/InstallonLinux
Windows: https://docs.splunk.com/Documentation/Splunk/7.3.0/Installation/ChoosetheuserSplunkshouldrunas
2. Configure it to send data to the old instance/indexer-to-be
https://docs.splunk.com/Documentation/Splunk/7.2.6/DistSearch/Forwardsearchheaddata
3. Configure it to use the old instance/indexer-to-be as a search-peer (same thing as indexer, different terminology)
https://docs.splunk.com/Documentation/Splunk/7.2.6/DistSearch/Configuredistributedsearch
4. Copy your apps from the old instance to the new search head (/opt/splunk/etc/apps)
App migration reference: https://docs.splunk.com/Documentation/Splunk/latest/DistSearch/Migratefromstandalonesearchheads
6. Restart your search head
7. Review .conf files in /opt/splunk/etc/system/local to determine what needs to be moved to the search head

That's a rough overview, but should get you to where you want to be.

[woodcock]

Do not use Windows for your infrastructure here.

[robertlynch2020]

noted we are on UNIX for everything

[woodcock]

You have saved yourself much avoidable pain.

[robertlynch2020]

Hi

Thanks for this, i will try it and get back

[robertlynch2020]

Hi

Thanks for this answer.
We have done the following. One new search head and 2 indexers. The 1st Indexer is the old production, but when we start it up all the datamodels start to rebuild. Is there a way to get the datamodels not to rebuild?

Thanks
Robert

[woodcock]

If you are going to use them, they must be rebuilt.

[robertlynch2020]

Ok thanks - that is disappointing to here, it looks like the migration process will take ~48 hour of MAX CPU on a 60 core machine.

We have ~20 Data models.

Lucky we are on a very very good machine, otherwise we would have to stop production or do a parallel run