Move indexed data partially to new indexer

[lukasmecir]

Hello,

I have all-in-one Splunk instance with data already indexed. Now I want to add new Indexer (not-clustered, clean installation). I would like to move part of indexed data to new Indexer (to have cca the same amount of data on both instances). My idea of process is:

1. Stop all-in-one instance 
2. Create new index(es) on new indexer
3. Stop new indexer
4. Copy (what is best - rsync?) part of buckets in given index(es) from all-in-one instance to new indexer
5. Start new indexer and all-in-one instance
6. Configure outputs.conf on forwarders - add new indexer
7. Add new indexer as search peer to all-in-one instance

Would it work or I missed something?

Thank you for help.

Best regards

Lukas Mecir

[PickleRick]

You can just set up a cluster with SF=RF=1 (mind you, that will not give you any redundancy) and have CM rebalance the buckets.

Hidden bonus - you don't have to manually track configs across indexers.

[lukasmecir]

Thank you for hint, sounds interesting, I will try. Redundancy is not desired in this case, so its no problem.

[gcusello]

Hi @lukasmecir ,

don't copy part of buckets, but divide your indexes between the two Indexers to have more or less the same amount od data, moving the entire Indexes, not part of them.

Remeber to add the same indexes.conf to the New Indexer.

Ciao.

Giuseppe

[lukasmecir]

Hi,

thank you for reply. I understand what you mean, but my intended goal is not only to have the same amount of data on both instances, but have the same amount of data on both instances per index - to share load between instances when data in one index will be searched. I see I did not mention this in original post, I am sorry for that, my fault.

[gcusello]

Hi @lukasmecir ,

I'm not sure that copying a part of buckets, an index will continue to correctly run: in theory it should do, but I'd prefer the approach I described.

Also because, in this way, you separate data to have more or less the same amunt of data in both the Indexers, and then the new data will be distributed between them.

You could try, waiting to delete copied buckets from the first Indexer after a test completion.

Ciao.

Giuseppe

[lukasmecir]

Hi,

I tried my process:

1. Clear install of new IDX
2. Run new IDX for the first time
3. Crate index on new IDX
4. Stop the new IDX
5. Stop the old all-in-one instance
6. Copy (by rsync -a command) desired WARM buckets (db_... dirs) from the old instance to new IDX
7. Delete copied buckets from old all-in-one instance
8. Start both instances
9. Add new IDX as search peer on the old instance
10. Reconfigure outputs.conf on forwarders to add new ID

Everything seems OK now, I let it running for some time and check again.

[isoutamo]

Hi
Every index has file which told last used bucket number. You should also update this to refer correct number on node where you have copied those buckets. Of course if you have copied whole indexes directory then you probably have copied also those files too. If you haven’t copy those them indexer could overwrite old buckets with new events.
r. Ismo

[gcusello]

Hi @lukasmecir ,

remember to copy indexes.conf on the new machines.

Ciao.

Giuseppe

[lukasmecir]

Yep, good point, thank you.

[gcusello]

Hi @lukasmecir ,

good for you, see next time!

let us know if we can help you more, or, please, accept one answer for the other people of Community.

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉