I have a single install (Everything on one machine).
I want to go to one search head and 2 indexers (non clustered) multiple machines.
Is there a set of instruction on how to do this, the doc is great but there seem to be so many options that I get lost.
I am looking for step 1 2, 3..etc...
I also have a question like:
If I change the current install from SH+Indexer -> indexer and create a separate search head(I think this is the best way),
do I have to reinstall all my apps onto the new search head?
There are plenty of Splunk documents on how to setup a distributed configuration (which you've probably encountered already):
New install: https://docs.splunk.com/Documentation/Splunk/7.2.6/DistSearch/Overviewofconfiguration
However, Google isn't really providing insight when migrating from a standalone to a distributed environment.
I think the easiest path would be as follows:
1. Install new Splunk instance (this will be the search head)
2. Configure it to send data to the old instance/indexer-to-be
3. Configure it to use the old instance/indexer-to-be as a search-peer (same thing as indexer, different terminology)
4. Copy your apps from the old instance to the new search head (/opt/splunk/etc/apps)
App migration reference: https://docs.splunk.com/Documentation/Splunk/latest/DistSearch/Migratefromstandalonesearchheads
6. Restart your search head
7. Review .conf files in /opt/splunk/etc/system/local to determine what needs to be moved to the search head
That's a rough overview, but should get you to where you want to be.
Thanks for this answer.
We have done the following. One new search head and 2 indexers. The 1st Indexer is the old production, but when we start it up all the datamodels start to rebuild. Is there a way to get the datamodels not to rebuild?
Ok thanks - that is disappointing to here, it looks like the migration process will take ~48 hour of MAX CPU on a 60 core machine.
We have ~20 Data models.
Lucky we are on a very very good machine, otherwise we would have to stop production or do a parallel run
If it was as easy as 1,2,3 the docs would say so. The docs have a lot of options because there are a lot of variables.
If I was in your shoes, I'd make the existing server an indexer and add new servers to act as search head and second indexer. Usually, it's not necessary to re-install apps - just transfer $SPLUNK_HOME/etc/apps from the old server to the new one. There are caveats so read the docs.
The second indexer will start out empty, but will accumulate data over time. Until then, however, searches won't benefit from the second indexer. Better is to cluster the indexes and balance the indexes from the start, but that's something for Professional Services to handle for you.