All Apps and Add-ons

Move Splunk Databases to new Indexer in new location

gnovak
Builder

I just got done reading these 2 articles but still am not sure I found what I"m looking for. Figured I'd put my question here and see if anyone else has done this

I read these:

http://docs.splunk.com/Documentation/Splunk/4.3.2/Admin/Moveanindex
http://splunk-base.splunk.com/answers/49533/movin-an-index-to-new-server

So here's my situation. A new datacenter has been acquired at the company I work for. Currently at this new datacenter, we have deployed 2 new Splunk Indexers. One is going to be used for data, another for network data. They aren't really doing too much right now.

The plan is that they want to take 2 other current splunk indexers from another datacenter in a totally different location and move all the contents from these 2 splunk indexers to the new one setup in the new datacenter.

In a nutshell:
-Move all contents of IndexerA and IndexerB located in Location1 to IndexerC located in Location2

Has anyone ever done this before? From what I'm reading it sounds like you just stop splunk and copy over the entire var/lib/splunk/defaultdb. I assume it's more complicated then this though.

Thoughts? Comments?

Tags (1)
1 Solution

richprescott
Path Finder

I've moved quite a few instances in the past and most of the time it is rather simple.

1 . Rolling hot buckets to warm For each of the databases that needs to be transferred, you will need to prepare the data for transfer. This can be done with the following command:

PathToSplunk\bin\splunk.exe _internal call /data/indexes/#DBname#/roll-hot-buckets -auth #un#:#pw#

2 . Physically move the index components to the new location. As the indexes can be very large, it is best to remote into either of the servers to perform the copy. If you are remoted into the source server, you can run the following command for each of the databases that needs to be moved:

copy PathToSplunk\var\lib\splunk\#dbname# \\NewPathToSplunk\var\lib\splunk\#dbname#

3 . Scrub the bucket IDs if necessary Advanced info for this topic.

4 . Point Splunk at the newly moved index. Modify the indexes.conf file to point at the location of the new database.

If you run into any issues, verify permissions on the copied files/folders.

View solution in original post

richprescott
Path Finder

I've moved quite a few instances in the past and most of the time it is rather simple.

1 . Rolling hot buckets to warm For each of the databases that needs to be transferred, you will need to prepare the data for transfer. This can be done with the following command:

PathToSplunk\bin\splunk.exe _internal call /data/indexes/#DBname#/roll-hot-buckets -auth #un#:#pw#

2 . Physically move the index components to the new location. As the indexes can be very large, it is best to remote into either of the servers to perform the copy. If you are remoted into the source server, you can run the following command for each of the databases that needs to be moved:

copy PathToSplunk\var\lib\splunk\#dbname# \\NewPathToSplunk\var\lib\splunk\#dbname#

3 . Scrub the bucket IDs if necessary Advanced info for this topic.

4 . Point Splunk at the newly moved index. Modify the indexes.conf file to point at the location of the new database.

If you run into any issues, verify permissions on the copied files/folders.

richprescott
Path Finder

Yes. For checking buckets, I wrote a quick PowerShell snippet that does a get-childitem and groups the directories based on fullname. If there are any groups larger than 1, then there is a bucket conflict. The same could probably be done in perl/python.

0 Karma

gnovak
Builder

One more thing, I assume I also have to stop splunk on the Indexer that I am moving these database files to as well, correct? Meaning, I can't move these database files to an indexer that currently has splunk running. I would assume you stop splunk, move files, look for bucket conflicts and fix, then restart. Correct?

0 Karma

richprescott
Path Finder

Yes, if you are using the forwarders, you will need to update outputs.conf to reflect the new servers/ports.

0 Karma

gnovak
Builder

I'm also wondering if these indexers are going to eventually go away (indexer A and B) won't you have to tell the forwarders to go to a new indexer with their data?

0 Karma

richprescott
Path Finder

Step 1 is performed while the server is still up. The rest are done while the service is stopped. Inputs.conf should not need to be modified. In my experiences, I have not had to scrub buckets, but there are certain circumstances where you would need to, which is why that step is included with a link to more info.

0 Karma

gnovak
Builder

I take it you do this when splunk is down. Also if you're basically moving the contents of the Main index from Indexer A and putting it also in the Main index on Indexer C, you don't really have to modify the inputs.conf. At least that's what I'm thinking. Would you say that's correct? Did you have to scrub any buckets at all?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...