topic Re: Move indexed data partially to new indexer in Installation

Move indexed data partially to new indexer

lukasmecir — Thu, 12 Dec 2024 09:11:46 GMT

Hello,

I have all-in-one Splunk instance with data already indexed. Now I want to add new Indexer (not-clustered, clean installation). I would like to move part of indexed data to new Indexer (to have cca the same amount of data on both instances). My idea of process is:

Stop all-in-one instance
Create new index(es) on new indexer
Stop new indexer
Copy (what is best - rsync?) part of buckets in given index(es) from all-in-one instance to new indexer
Start new indexer and all-in-one instance
Configure outputs.conf on forwarders - add new indexer
Add new indexer as search peer to all-in-one instance

Would it work or I missed something?

Thank you for help.

Best regards

Lukas Mecir

Re: Move indexed data partially to new indexer

gcusello — Thu, 12 Dec 2024 09:54:18 GMT

Hi @lukasmecir ,

don't copy part of buckets, but divide your indexes between the two Indexers to have more or less the same amount od data, moving the entire Indexes, not part of them.

Remeber to add the same indexes.conf to the New Indexer.

Ciao.

Giuseppe

Re: Move indexed data partially to new indexer

lukasmecir — Thu, 12 Dec 2024 10:06:04 GMT

Hi,

thank you for reply. I understand what you mean, but my intended goal is not only to have the same amount of data on both instances, but have the same amount of data on both instances per index - to share load between instances when data in one index will be searched. I see I did not mention this in original post, I am sorry for that, my fault.

Re: Move indexed data partially to new indexer

gcusello — Thu, 12 Dec 2024 10:16:58 GMT

Hi @lukasmecir ,

I'm not sure that copying a part of buckets, an index will continue to correctly run: in theory it should do, but I'd prefer the approach I described.

Also because, in this way, you separate data to have more or less the same amunt of data in both the Indexers, and then the new data will be distributed between them.

You could try, waiting to delete copied buckets from the first Indexer after a test completion.

Ciao.

Giuseppe

Re: Move indexed data partially to new indexer

PickleRick — Thu, 12 Dec 2024 13:54:09 GMT

You can just set up a cluster with SF=RF=1 (mind you, that will not give you any redundancy) and have CM rebalance the buckets.

Hidden bonus - you don't have to manually track configs across indexers.

Re: Move indexed data partially to new indexer

lukasmecir — Thu, 12 Dec 2024 14:54:42 GMT

Thank you for hint, sounds interesting, I will try. Redundancy is not desired in this case, so its no problem.

Re: Move indexed data partially to new indexer

lukasmecir — Thu, 12 Dec 2024 15:00:48 GMT

Hi,

I tried my process:

Clear install of new IDX
Run new IDX for the first time
Crate index on new IDX
Stop the new IDX
Stop the old all-in-one instance
Copy (by rsync -a command) desired WARM buckets (db_... dirs) from the old instance to new IDX
Delete copied buckets from old all-in-one instance
Start both instances
Add new IDX as search peer on the old instance
Reconfigure outputs.conf on forwarders to add new ID

Everything seems OK now, I let it running for some time and check again.

Re: Move indexed data partially to new indexer

gcusello — Thu, 12 Dec 2024 15:08:46 GMT

Hi @lukasmecir ,

remember to copy indexes.conf on the new machines.

Ciao.

Giuseppe

Re: Move indexed data partially to new indexer

lukasmecir — Thu, 12 Dec 2024 15:21:26 GMT

Yep, good point, thank you.

Re: Move indexed data partially to new indexer

gcusello — Thu, 12 Dec 2024 15:27:18 GMT

Hi @lukasmecir ,

good for you, see next time!

let us know if we can help you more, or, please, accept one answer for the other people of Community.

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉

Re: Move indexed data partially to new indexer

isoutamo — Thu, 12 Dec 2024 17:52:45 GMT

Hi
Every index has file which told last used bucket number. You should also update this to refer correct number on node where you have copied those buckets. Of course if you have copied whole indexes directory then you probably have copied also those files too. If you haven’t copy those them indexer could overwrite old buckets with new events.
r. Ismo