How do I migrate data from a single install (search head + indexer) to a search head + X indexers?

[robertlynch2020]

Hi

I have worked out how to set up a distributed search on my beta site, but how do i move the current data "indexes" and "datamodels" off the search head to a new indexer?

Current = One search head + one indexer on the same install
Future = One search head + X indexers (New installs) + Original Indexer(With original data).

As all my APPs are installed on the current install, i think it is best if I keep this search head and move the data to a new indexer(and put in the old data), then add on X new empty indexers - Right?

So do I set up another Splunk install and copy over files from the original install to populate it? If so what files?

/splunk/var/lib/splunk, perhaps, is this folder or maybe a subset? [How come i cant find doc on this? i can't be the first person to do this right?]

Thanks in advance

Robbie

[woodcock · Accepted solution]

First, go with indexer clustering so that you can do a rebalance of the old data. Copy $SPLUNK_HOME/etc/apps from the original server to the new Search Head. Setup the new Search Head with an outputs.conf to forward its logs to the Indexers. Disable local login to ALL Indexers. Initiate data rebalance on the Indexers from the Cluster Master.

BE SURE TO READ THE COMMENTARY BELOW THIS ANSWER!

[gpullis]

I'm doing the same thing, basically. Moving from a single Splunk server to a separate search head and indexer. I copied $SPLUNK_HOME/etc/apps and $SPLUNK_HOME/etc/users to my new search head.

What's missing is my authentication config. What do I need to copy to get that?

[woodcock]

It should be in authentication.conf and authorize.conf. If you used the GUI, this will be in $SPLUNK_HOME/etc/system/local/. You can see for sure by running this:

find $SPLUNK_HOME -name "auth*.conf"

[gpullis]

Thanks! That got me my LDAP config.

I did have to manually re-enter the password for my LDAP binding account, but otherwise that's progress!Oh, and the fun part was trying to get the password saved faster than the search head could get the account locked out. 🙂

Now, how do I replicate local accounts?

Thanks again.

[woodcock]

Local accounts should be in $SPLUNK_HOME/etc/passwd but if your new server did not get seeded with the splunk.secret file from the old server, then all the passwords will need to be reset once you copy this file.

[gpullis]

Thanks. Lucky for me, I just have a few local accounts used for ops wall mount displays, and their passwords are documented. This would be a bummer for someone with a lot of local accounts.

If Splunk ever gets around to writing a guide for how to expand out of a single instance into multiple servers, splunk.secret would be a super important thing to mention early on.

The other thing is that, when you copy your $SPLUNK_HOME/etc/apps all your inputs come along for the ride, and you want to give some early thought as to where your inputs are running when the dust settles.

[woodcock]

I agree, I should have mentioned it earlier.

[rupeshn]

@woodcock

Similar type of situation, But my standalone indexer which acts as SH, Indexer and DS has very few(2-3) Apps. So I plan to delete those apps first, then set it as indexer and add 2 new indexers and add new SH. Would that be feasible? If so, what are Pros and cons involved. Please provide your inputs.

[woodcock]

Tje very first thing that I would do in your situation is move the DS, if that is changing. The problem with that is that most people who do not know better use the CLI and set deploy-server to setup Deployment Clients instead of dropping a DeploymentClient app in $SPLUNK_HOME/etc/apps/. Make sure that when you update your DCs that you fix this mistake and use an app.

[robertlynch2020]

HI

Cheers and thanks, i will give this a go. 🙂 This is how i was thinking i should give it a go.
It was the getting my apps over bit i was not 100% sure, but it makes sence that i can just copy them - cool cool cool.

Robbie

[robertlynch2020]

One question.

Do i need to install the apps onto the indexers(or part of them), or will index clustering take care of index creation and datamodel creation on the new indexers?

Rob

[woodcock]

Be sure to copy the splunk.secret file from the old server to the new servers BEFORE you start the new servers. That way, any encrypted passwords stored will still work. Generally it is safe to install everything everywhere so, yes, install the apps on the Indexers. Some of them will be necessary and some of them won't but the ones that aren't will be harmless.

[robertlynch2020]

Cheers my man 🙂

[woodcock]

Also, do not keep the original indexer if the new indexers are beefier in any significant way. Every search is only as fast as the WORST indexer. Having 99 blazing fast indexers and 1 slow one will be pretty much the same as having 100 slow indexers.

[kml_uvce]

move your current search head(etc/apps) to new search head , and change current (search head + indexer) as indexer

[robertlynch2020]

Thanks for the answer 🙂

[robertlynch2020]

Hi

I was looking at the doc and it looks like it should move my current install to an indexer and reimport my apps to a new search head.

SO I will try this and get back with updates

Rob