topic Re: Move index and resize in Getting Data In

Move index and resize

jrich523 — Wed, 13 Nov 2013 17:48:44 GMT

I have an Index which is on NAS. Im mostly having connection issues (win 2012R2 and Isilon NAS)

so i was thinking about moving it local, restricting its size and using the NAS as Cold storage however i have a small problem that i dont want to figure out by testing.

The index i'd like to move is currently about 200GB of data, I'd like to limit the warm on the new location (local to the server) to 50gb.

so if i change the config file and move the files, when it starts up will it start to push that extra 150gb of files over ok?

Re: Move index and resize

lguinn2 — Thu, 14 Nov 2013 06:33:51 GMT

So, for my explanation, I am going to call your local drive 😧 and the NAS drive N:, and assume that the index is named myIndex

I would do it this way: set up separate directories on 😧 & N: for the various parts of the indexes, copy in the data, edit indexes.conf

1) Stop Splunk

2) Create the directories on each drive (they can be named anything, actually) -

D:\Splunk\myIndex\db

D:\Splunk\myIndex\thaweddb

N:\Splunk\myIndex\colddb

3) Find the indexes.conf that currently contains the specifications for myIndex. Make a backup copy of this file. Note the current locations of each directory in the file.

4) Edit the following lines in 'indexes.conf'. This will set the new locations - and restrict the size of the hot/warm directory (db). Notice that part of the homepath space will be used by the hot buckets, but you can increase the size to account for that, of course.

[myIndex]
homePath = D:\Splunk\myIndex\db
coldPath = N:\Splunk\myIndex\colddb
thawedPath = D:\Splunk\myIndex\thaweddb
homePath.maxDataSizeMB = 50000

5) Backup the current index. Copy/move the files from each of the old db, colddb and thaweddb directories into the new locations. Be careful to maintain the file attributes (ownership, permissions, etc) and to copy the whole directory tree.

6) Restart Splunk.

Everything should now be in its proper place and the new limitations on the size will take effect immediately, which means that Splunk will roll the warm buckets as quickly as possible to comply. You might be able to manually move some warm buckets to cold - but you would have to carefully choose the buckets and it would probably be just as quick to let Splunk do it.

Re: Move index and resize

jrich523 — Thu, 14 Nov 2013 18:45:49 GMT

Thanks Iguinn,
that worked ok with the exception of the limitation on the size. its currently at 150gb (original moved size) and hasnt gone down at all since i moved it yesterday.

im investigating the splunkd.log now to see if i can pinpoint anything.

Re: Move index and resize

lguinn2 — Thu, 14 Nov 2013 20:09:13 GMT

I'd check permissions on the directories, too. And another setting that might be helpful:

maxWarmDBCount = 70

This says that the maximum number of warm buckets is 70; if your default bucket size is 750 MB, this should work out to about 50 GB.

Re: Move index and resize

sowings — Thu, 14 Nov 2013 20:18:27 GMT

Note that many times, for high volume indexes, the max size isn't met, so scaling the usage of the home path strictly by warm bucket count isn't always straightforward.