How to migrate a whole indexer with almost no logs lost ?

[ctaf]

Hello,

I am working on a migration process and I would love your opinion on the solution I've been think to.
My environment: 1 search head, 3 indexers (non-clustered, they store different data), 1 deployment server, about 200 universal forwarders

I have to move each 3 indexers on 3 new servers. Of course, I would like to loose the less data indexed as possible. Here is what I would do:

1- Copy the Splunk folder of the old Indexer servers onto the new Indexer servers (without the Database folder)
2- On the Universal Forwarders, change (with the help of the Deployment server) the IP address where to forwards the data to: the new servers. Check "Auto Restart" in the server class.
3- When all the new data is being redirected to the new Servers, roll the hot bucket of the old servers to warm state. Then change the localid of the buckets to be greater than 10 (in order to be greater than the localid of the new Indexers)
4- Copy those old buckets onto the new servers. The hot bucket will probably have a localid of "1" whereas the old buckets start at "10"

Note : If the ID of older data is greater than ID of newer data, does that work? The next hot bucket will not be the "2", right?

What do you think about this solution?

Thank you

[esix_splunk · Accepted solution]

This is general works, but I think you're making it a bit complex. Since this isnt a multisite cluster, you can copy the buckets between servers and they will be searchable. Here's a general migration pathway that you can consider:

1) Build and deploy your new servers, configure them to index
2) Setup distributed search from your search head to include the new indexers
3) On your deployment server, decrease your phonehome interval to a low enough rate that it wont overload / tax your DS (200 servers isnt too much, 90 seconds? 5 minutes would be fine..)
4) Change your deployment outputs, point the clients to the new indexers and redeploy.
5) Confirm no new data is coming on on your old indexers and you can search all 6 indexers
6) Once all forwarders are migrated, roll the buckets on the old indexers, start the copy of the warm/cold/frozen buckets to the respective indexers
7) Restart those indexers and confirm the buckets are searchable ( If you have some bucket collisions, you can change the id of the bucket)
8) Disable distributed search on the old peers
9) Decommission the old peers

Have done this many times. Please let me know if you want any more details on this and parts of this.

[ctaf]

Hello Esix,

Thank you for your help

Instead of steps 1 and 2, I will probably just copy the /etc/ folder.
But this brings me another question: One of these indexer is my licence master. So at step 5), I will have 2 licence master. I am thinking it will bring some issues..

Moreover, Instead of step 3, I can just force a deploy?

[Jeremiah]

Here is what we've done in the past. Lets say you have oldserver1-3 and newserver1-3.

First, rsync all of the indexed data from the old servers to the new servers, excluding the hot buckets. Depending on how much data you have, this may take a few days.

When your rsync is complete. Remove oldserver1 from the forwarder configurations so that it is no longer receiving data. Once it is no longer receiving data, restart it. Do a final rsync to newserver1. Once the final rsync is complete, add newserver1 to the forwarder's configuration. Repeat this process for each pair of indexers.

This process works well if you are using auto-load balancing for your forwarder connections to your indexers, and that you are able to run on two indexers only for the the time it take to do the indexer cutover.

[ctaf]

Anyone, please?

[jmallorquin]

Hi ctaf

What kind of sources do you index, becouse if all the sources are log, maybe you can stop all the indexes, move to the new ones, point all the forwarders to the new indexers and start the new indexers.

Hope i help you

[ctaf]

It is just logs being forwarded by universal forwarders. There is no pause in the logs as they keeps being generated all the time.
If I just stop the old indexers, I will loose data during the migration.

[jmallorquin]

Hi ctaf,

If you stop the indexers, the UF will stop send events. When you start the indexers the UF will send from the point where was the UF when you stop the indexer IF the log still available for the UF, so you will not loss events.

[ctaf]

I have a lot of data, I am afraid that the buffer will soon be overloaded. What about my solution of changing the IP address of the Indexer on the UF in the outputs.conf? Then, very quickly, all the data will be directed to the new server.
I will not have the time to copy all the indexed data to the new server and turning it on again.

[jmallorquin]

Hi ctaf,

The buffer of the UF are used only in TCP inputs, no for logs inputs.
I am no sure that move the data in bucket level is the best way, maybe in the future buckets have the same id...

I preffer to analyze the source and if all are logs that will exist during the time of the migration, I stoped all copy to the new servers, point all forwarders to the new and start the indexers. With this you will not have event loss.

Hope i help you.

[ctaf]

Oh OK, I understand what you meant with TCP/logs inputs.

Thank you for your help.
Now if anyone know about the localid order important, I'd love to know too, just for curiousity.