How to move Splunk db to a new location?

[lakromani]

I ran out of disk space, so I added a new disk to my Ubuntu system.
Problem is to move the db from old to new location.

root is running the Splunk daemon.
root 16938 3.2 2.4 489816 199604 ? Sl 18:04 6:36 splunkd -p 8089 start
Splunk 6.5.2 is installed in /opt/splunk
I would like move db to /mnt/sdb1/splunkdb
Disk and directory are ok.
Then I tried to follow the instruction here:
https://docs.splunk.com/Documentation/Splunk/6.5.2/Indexer/Moveanindex
Here thinks goes rapidly wrong.
When I run cp -rp $SPLUNK_DB/* /mnt/sdb1/splunkdb, it did not just copy the Splunk DB, but the whole disk....
Why? echo "$SPLUNK_DB" does not give anything, so when I was at root folder, it copied all files.

So what do I do wrong?

PS did look for other solutions, but none give me any good result.

[javierop]

If you want to know the value of $SPLUNK_DB that Splunk is using, you can use this command

`ps -ex | grep --color SPLUNK_DB`

then you can declare the same var in your enviroment

export SPLUNK_DB=/opt/splunk/var/lib/splunk

and finally copy the files .

As you posted, in this doc you cand find the file were SPLUNK_DB is defined.

[woodcock · Accepted solution]

It is really very easy. Assuming that you are moving every index:

Figure out where your DBs are now; the default is $SPLUNK_DB and the default for this is /opt/splunk/var/lib/ so we will go with that.
Find the indexes.conf file that defines this on your indexers (if linux, you can use this from the CLI: find / -name indexes.conf).
Stop splunk on your indexers with ./splunk stop.
Edit your indexes.conf file to point every DB of every index to the new location (i.e. global search and replace of $SPLUNK_DB/ to /mnt/sdb1/splunkdb/
Move the entire set of stuff with something like: cp -rp /opt/splunk/var/lib/* /mnt/sdb1/splunkdb/
Restart splunk with ./splunk start

Your problem was surely that you did not have the $SPLUNK_DB environment variable set, which on linux you can do temporarily with export SPLUNK_DB=/opt/splunk/var/lib/. Without this set, it was essentially interpreted as ./ which is not a valid path for those files.

[lakromani]

This was close to what I have done.

Only difference is that I changed /opt/splunk/etc/splunk-launch.conf instead of /opt/splunk/etc/apps/search/local/indexes.conf
In /opt/splunk/etc/splunk-launch.conf I added SPLUNK_DB=/mnt/sdb1/splunk-db/
This will then change the variable of the path $SPLUNK_DB instead of hard coding the path.

I will mark your as accepted since it close to what I have done.

[somesoni2]

Check for value of $SPLUNK_DB in file $SPLUNK_HOME/etc/splunk-launch.conf. Then instead of using environment variable, use the actual path in your copy command.

[lakromani]

Seems to be this folder /opt/splunk/var/lib/splunk. So I will try to copy all files and change /opt/splunk/etc/splunk-launch.conf and see how it goes.

[ddrillic]

Right - /opt/splunk/var/lib/splunk is the default location of the indexes.

[lakromani]

So I should change this:
# SPLUNK_DB=/home/build/build-home/ember/var/lib/splunk
to
SPLUNK_DB=/mnt/sdb1/splunkdb

I am still not sure what folder to copy files from to get my index db.
It's a plain install of splunk in /opt/splunk folder

PS $SPLUNK_HOME does not contain any variable, so i read file /opt/splunk/etc/splunk-launch.conf

[somesoni2]

You can go to Splunk UI of the Splunk servers (indexer I believe), go to Settings->Indexes in there, and see the HomePath column to see the actual path were index data is stored. From that path everything before <indexname>/db is the directory where all index data is stored. So, if the path says (example) /opt/splunk/var/lib/splunk/myindex/db (where myindex is the index name), then /opt/splunk/var/lib/splunk/ is your $SPLULNK_DB and you should be copying everything from /opt/splunk/var/lib/splunk/* to new drive. If its /some/other/path/myindex/db, then /some/other/path is your $SPLULNK_DB and you should be copying everything from /some/other/path/* to new drive.