Move Splunk Databases to new Indexer in new location

[gnovak]

I just got done reading these 2 articles but still am not sure I found what I"m looking for. Figured I'd put my question here and see if anyone else has done this

I read these:

http://docs.splunk.com/Documentation/Splunk/4.3.2/Admin/Moveanindex
http://splunk-base.splunk.com/answers/49533/movin-an-index-to-new-server

So here's my situation. A new datacenter has been acquired at the company I work for. Currently at this new datacenter, we have deployed 2 new Splunk Indexers. One is going to be used for data, another for network data. They aren't really doing too much right now.

The plan is that they want to take 2 other current splunk indexers from another datacenter in a totally different location and move all the contents from these 2 splunk indexers to the new one setup in the new datacenter.

In a nutshell:
-Move all contents of IndexerA and IndexerB located in Location1 to IndexerC located in Location2

Has anyone ever done this before? From what I'm reading it sounds like you just stop splunk and copy over the entire var/lib/splunk/defaultdb. I assume it's more complicated then this though.

Thoughts? Comments?

[richprescott · Accepted solution]

I've moved quite a few instances in the past and most of the time it is rather simple.

1 . Rolling hot buckets to warm For each of the databases that needs to be transferred, you will need to prepare the data for transfer. This can be done with the following command:

PathToSplunk\bin\splunk.exe _internal call /data/indexes/#DBname#/roll-hot-buckets -auth #un#:#pw#

2 . Physically move the index components to the new location. As the indexes can be very large, it is best to remote into either of the servers to perform the copy. If you are remoted into the source server, you can run the following command for each of the databases that needs to be moved:

copy PathToSplunk\var\lib\splunk\#dbname# \\NewPathToSplunk\var\lib\splunk\#dbname#

3 . Scrub the bucket IDs if necessary Advanced info for this topic.

4 . Point Splunk at the newly moved index. Modify the indexes.conf file to point at the location of the new database.

If you run into any issues, verify permissions on the copied files/folders.

[richprescott]

Yes. For checking buckets, I wrote a quick PowerShell snippet that does a get-childitem and groups the directories based on fullname. If there are any groups larger than 1, then there is a bucket conflict. The same could probably be done in perl/python.

[gnovak]

One more thing, I assume I also have to stop splunk on the Indexer that I am moving these database files to as well, correct? Meaning, I can't move these database files to an indexer that currently has splunk running. I would assume you stop splunk, move files, look for bucket conflicts and fix, then restart. Correct?

[richprescott]

Yes, if you are using the forwarders, you will need to update outputs.conf to reflect the new servers/ports.

[gnovak]

I'm also wondering if these indexers are going to eventually go away (indexer A and B) won't you have to tell the forwarders to go to a new indexer with their data?

[richprescott]

Step 1 is performed while the server is still up. The rest are done while the service is stopped. Inputs.conf should not need to be modified. In my experiences, I have not had to scrub buckets, but there are certain circumstances where you would need to, which is why that step is included with a link to more info.

[gnovak]

I take it you do this when splunk is down. Also if you're basically moving the contents of the Main index from Indexer A and putting it also in the Main index on Indexer C, you don't really have to modify the inputs.conf. At least that's what I'm thinking. Would you say that's correct? Did you have to scrub any buckets at all?