topic Re: Is it possible to migrate indexed buckets to a different index on a new machine? in Installation

Is it possible to migrate indexed buckets to a different index on a new machine?

Glenn — Mon, 28 Sep 2020 09:58:56 GMT

I have seen an answer to part of my question (whether I can copy the contents of an index to another server) in the answer below:

http://splunk-base.splunk.com/answers/3516/how-do-i-migrate-my-splunk-data-to-a-new-machine

But I have a further query on this: Is it possible to migrate the data from an old server under a certain index (eg. defaultdb) to a new server into a different index (say eg. migrateddb)?

Does the indexed data in the buckets (eg. /opt/splunk/var/lib/splunk/defaultdb/db/db_1318590087_1318108850_7491) contain anything that ties them to that specific defaultdb index name?

Re: Is it possible to migrate indexed buckets to a different index on a new machine?

Drainy — Thu, 13 Oct 2011 13:39:32 GMT

I would suspect not but I'm intrigued to know now - hadn't thought of that before 🙂

Re: Is it possible to migrate indexed buckets to a different index on a new machine?

yannK — Mon, 28 Sep 2020 09:58:58 GMT

There is no information about the index in the buckets, you can migrate a bucket manually to a :

new indexer
new index folder

Recommendations are :

roll your hot buckets to warm before (easier to manipulate after)
When merging buckets in the same folder, check the buckets ID of the existing and imported buckets to avoid duplicates ID. the ID is the last number of each bucket folder : db_xxx_xxx_ or hot_v1_ you can re-enumerate the IDs of the new buckets at the end of the existing range, or in a completely new range (like 100 units further)
don't migrate 64bit to 32bit, some buckets may be larger that the system file size capacity

Re: Is it possible to migrate indexed buckets to a different index on a new machine?

Glenn — Thu, 13 Oct 2011 15:06:19 GMT

Just the answer I was looking for, cheers.

Re: Is it possible to migrate indexed buckets to a different index on a new machine?

yannK — Mon, 28 Sep 2020 12:55:05 GMT

Some extra details on the migration of indexes to a new indexer.

If you just want to move your indexed data from the old standalone indexer to a new indexer cluster, this is an easy procedure of copying files.

Usually the indexes to migrate are :
- main ($SPLUNK_HOME/var/lib/splunk/defaultdb on the disk by default)
- summary ($SPLUNK_HOME/var/lib/splunk/summary on the disk by default)
and any other custom indexes you may have.

first create the indexes on the new indexers and start splunk once (to create the folders)

For each index you want to migrate, you will find buckets folders in the /db and /colddb folders that you will have to copy to the new location (or distributed them one by one on a different indexer to distribute the volume).
Bucket folder names are db__ or hot_v1, the last number is an unique id, that increments. In each index, all buckets must have a unique ID.

You may not need to do anything if the destination indexes are empty. But if you are merging buckets from indexes, or splitting then over several indexers, you want to avoid duplicated of the id. The easy way is to increment the id in a range higher than the existing ones.

In case of duplicate id, a warming will be recorded in splunkd.log at start and the index will be disabled. If it happens, just fix the id, re-enable the index and start/

Re: Is it possible to migrate indexed buckets to a different index on a new machine?

vasanthmss — Thu, 12 Mar 2015 21:51:23 GMT

Check this,

http://wiki.splunk.com/Community:MoveIndexes

Re: Is it possible to migrate indexed buckets to a different index on a new machine?

jaxjohnny2000 — Mon, 14 Jan 2019 17:38:22 GMT

That's the easy part. The hard part is moving from one index cluster to another.

How would we move indexed data from a large production index cluster to a smaller development index cluster. The idea here is to be able to make changes in a lower environment before modifying production. Especially with Enterprise Security, we would make changes, see the impact, (e.g. did it work, what did it do to the indexer host, etc), then copy those changes to production.

So we would want to copy indexes from one cluster to another, but only a subset; maybe 90 days worth?

Is there a documented process for this/

Re: Is it possible to migrate indexed buckets to a different index on a new machine?

coreyCLI — Fri, 17 Mar 2023 15:46:13 GMT

@jaxjohnny2000 Did you ever get clarity on your issue? I am have the same problem. I need to migrate about 8 months worth of data (buckets) from one indexer cluster to another indexer cluster.

Re: Is it possible to migrate indexed buckets to a different index on a new machine?

PickleRick — Thu, 23 Mar 2023 18:28:48 GMT

I don't know if you noticed but this thread is already 12 years old and the post you're asking about is 4 years old. You have better chance of getting a response by writing a new question (maybe pointing to this old thread for reference) instead of digging up such ancient history 😉