Installation

How to move Splunk db to a new location?

lakromani
Builder

I ran out of disk space, so I added a new disk to my Ubuntu system.
Problem is to move the db from old to new location.

root is running the Splunk daemon.
root 16938 3.2 2.4 489816 199604 ? Sl 18:04 6:36 splunkd -p 8089 start
Splunk 6.5.2 is installed in /opt/splunk
I would like move db to /mnt/sdb1/splunkdb
Disk and directory are ok.
Then I tried to follow the instruction here:
https://docs.splunk.com/Documentation/Splunk/6.5.2/Indexer/Moveanindex
Here thinks goes rapidly wrong.
When I run cp -rp $SPLUNK_DB/* /mnt/sdb1/splunkdb, it did not just copy the Splunk DB, but the whole disk....
Why? echo "$SPLUNK_DB" does not give anything, so when I was at root folder, it copied all files.

So what do I do wrong?

PS did look for other solutions, but none give me any good result.

Labels (1)
0 Karma
1 Solution

woodcock
Esteemed Legend

It is really very easy. Assuming that you are moving every index:

Figure out where your DBs are now; the default is $SPLUNK_DB and the default for this is /opt/splunk/var/lib/ so we will go with that.
Find the indexes.conf file that defines this on your indexers (if linux, you can use this from the CLI: find / -name indexes.conf).
Stop splunk on your indexers with ./splunk stop.
Edit your indexes.conf file to point every DB of every index to the new location (i.e. global search and replace of $SPLUNK_DB/ to /mnt/sdb1/splunkdb/
Move the entire set of stuff with something like: cp -rp /opt/splunk/var/lib/* /mnt/sdb1/splunkdb/
Restart splunk with ./splunk start

Your problem was surely that you did not have the $SPLUNK_DB environment variable set, which on linux you can do temporarily with export SPLUNK_DB=/opt/splunk/var/lib/. Without this set, it was essentially interpreted as ./ which is not a valid path for those files.

View solution in original post

javierop
Engager

If you want to know the value of $SPLUNK_DB that Splunk is using, you can use this command

`ps -ex | grep --color SPLUNK_DB`

then you can declare the same var in your enviroment

export SPLUNK_DB=/opt/splunk/var/lib/splunk

and finally copy the files .

As you posted, in this doc you cand find the file were SPLUNK_DB is defined.

0 Karma

woodcock
Esteemed Legend

It is really very easy. Assuming that you are moving every index:

Figure out where your DBs are now; the default is $SPLUNK_DB and the default for this is /opt/splunk/var/lib/ so we will go with that.
Find the indexes.conf file that defines this on your indexers (if linux, you can use this from the CLI: find / -name indexes.conf).
Stop splunk on your indexers with ./splunk stop.
Edit your indexes.conf file to point every DB of every index to the new location (i.e. global search and replace of $SPLUNK_DB/ to /mnt/sdb1/splunkdb/
Move the entire set of stuff with something like: cp -rp /opt/splunk/var/lib/* /mnt/sdb1/splunkdb/
Restart splunk with ./splunk start

Your problem was surely that you did not have the $SPLUNK_DB environment variable set, which on linux you can do temporarily with export SPLUNK_DB=/opt/splunk/var/lib/. Without this set, it was essentially interpreted as ./ which is not a valid path for those files.

lakromani
Builder

This was close to what I have done.

Only difference is that I changed /opt/splunk/etc/splunk-launch.conf instead of /opt/splunk/etc/apps/search/local/indexes.conf
In /opt/splunk/etc/splunk-launch.conf I added SPLUNK_DB=/mnt/sdb1/splunk-db/
This will then change the variable of the path $SPLUNK_DB instead of hard coding the path.

I will mark your as accepted since it close to what I have done.

0 Karma

somesoni2
Revered Legend

Check for value of $SPLUNK_DB in file $SPLUNK_HOME/etc/splunk-launch.conf. Then instead of using environment variable, use the actual path in your copy command.

lakromani
Builder

Seems to be this folder /opt/splunk/var/lib/splunk. So I will try to copy all files and change /opt/splunk/etc/splunk-launch.conf and see how it goes.

0 Karma

ddrillic
Ultra Champion

Right - /opt/splunk/var/lib/splunk is the default location of the indexes.

0 Karma

lakromani
Builder

So I should change this:
# SPLUNK_DB=/home/build/build-home/ember/var/lib/splunk
to
SPLUNK_DB=/mnt/sdb1/splunkdb

I am still not sure what folder to copy files from to get my index db.
It's a plain install of splunk in /opt/splunk folder

PS $SPLUNK_HOME does not contain any variable, so i read file /opt/splunk/etc/splunk-launch.conf

0 Karma

somesoni2
Revered Legend

You can go to Splunk UI of the Splunk servers (indexer I believe), go to Settings->Indexes in there, and see the HomePath column to see the actual path were index data is stored. From that path everything before <indexname>/db is the directory where all index data is stored. So, if the path says (example) /opt/splunk/var/lib/splunk/myindex/db (where myindex is the index name), then /opt/splunk/var/lib/splunk/ is your $SPLULNK_DB and you should be copying everything from /opt/splunk/var/lib/splunk/* to new drive. If its /some/other/path/myindex/db, then /some/other/path is your $SPLULNK_DB and you should be copying everything from /some/other/path/* to new drive.

0 Karma
Get Updates on the Splunk Community!

Announcing General Availability of Splunk Incident Intelligence!

Digital transformation is real! Across industries, companies big and small are going through rapid digital ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...