Installation

How to move Splunk db to a new location?

lakromani
Builder

I ran out of disk space, so I added a new disk to my Ubuntu system.
Problem is to move the db from old to new location.

root is running the Splunk daemon.
root 16938 3.2 2.4 489816 199604 ? Sl 18:04 6:36 splunkd -p 8089 start
Splunk 6.5.2 is installed in /opt/splunk
I would like move db to /mnt/sdb1/splunkdb
Disk and directory are ok.
Then I tried to follow the instruction here:
https://docs.splunk.com/Documentation/Splunk/6.5.2/Indexer/Moveanindex
Here thinks goes rapidly wrong.
When I run cp -rp $SPLUNK_DB/* /mnt/sdb1/splunkdb, it did not just copy the Splunk DB, but the whole disk....
Why? echo "$SPLUNK_DB" does not give anything, so when I was at root folder, it copied all files.

So what do I do wrong?

PS did look for other solutions, but none give me any good result.

Labels (1)
0 Karma
1 Solution

woodcock
Esteemed Legend

It is really very easy. Assuming that you are moving every index:

Figure out where your DBs are now; the default is $SPLUNK_DB and the default for this is /opt/splunk/var/lib/ so we will go with that.
Find the indexes.conf file that defines this on your indexers (if linux, you can use this from the CLI: find / -name indexes.conf).
Stop splunk on your indexers with ./splunk stop.
Edit your indexes.conf file to point every DB of every index to the new location (i.e. global search and replace of $SPLUNK_DB/ to /mnt/sdb1/splunkdb/
Move the entire set of stuff with something like: cp -rp /opt/splunk/var/lib/* /mnt/sdb1/splunkdb/
Restart splunk with ./splunk start

Your problem was surely that you did not have the $SPLUNK_DB environment variable set, which on linux you can do temporarily with export SPLUNK_DB=/opt/splunk/var/lib/. Without this set, it was essentially interpreted as ./ which is not a valid path for those files.

View solution in original post

javierop
Engager

If you want to know the value of $SPLUNK_DB that Splunk is using, you can use this command

`ps -ex | grep --color SPLUNK_DB`

then you can declare the same var in your enviroment

export SPLUNK_DB=/opt/splunk/var/lib/splunk

and finally copy the files .

As you posted, in this doc you cand find the file were SPLUNK_DB is defined.

0 Karma

woodcock
Esteemed Legend

It is really very easy. Assuming that you are moving every index:

Figure out where your DBs are now; the default is $SPLUNK_DB and the default for this is /opt/splunk/var/lib/ so we will go with that.
Find the indexes.conf file that defines this on your indexers (if linux, you can use this from the CLI: find / -name indexes.conf).
Stop splunk on your indexers with ./splunk stop.
Edit your indexes.conf file to point every DB of every index to the new location (i.e. global search and replace of $SPLUNK_DB/ to /mnt/sdb1/splunkdb/
Move the entire set of stuff with something like: cp -rp /opt/splunk/var/lib/* /mnt/sdb1/splunkdb/
Restart splunk with ./splunk start

Your problem was surely that you did not have the $SPLUNK_DB environment variable set, which on linux you can do temporarily with export SPLUNK_DB=/opt/splunk/var/lib/. Without this set, it was essentially interpreted as ./ which is not a valid path for those files.

lakromani
Builder

This was close to what I have done.

Only difference is that I changed /opt/splunk/etc/splunk-launch.conf instead of /opt/splunk/etc/apps/search/local/indexes.conf
In /opt/splunk/etc/splunk-launch.conf I added SPLUNK_DB=/mnt/sdb1/splunk-db/
This will then change the variable of the path $SPLUNK_DB instead of hard coding the path.

I will mark your as accepted since it close to what I have done.

0 Karma

somesoni2
Revered Legend

Check for value of $SPLUNK_DB in file $SPLUNK_HOME/etc/splunk-launch.conf. Then instead of using environment variable, use the actual path in your copy command.

lakromani
Builder

Seems to be this folder /opt/splunk/var/lib/splunk. So I will try to copy all files and change /opt/splunk/etc/splunk-launch.conf and see how it goes.

0 Karma

ddrillic
Ultra Champion

Right - /opt/splunk/var/lib/splunk is the default location of the indexes.

0 Karma

lakromani
Builder

So I should change this:
# SPLUNK_DB=/home/build/build-home/ember/var/lib/splunk
to
SPLUNK_DB=/mnt/sdb1/splunkdb

I am still not sure what folder to copy files from to get my index db.
It's a plain install of splunk in /opt/splunk folder

PS $SPLUNK_HOME does not contain any variable, so i read file /opt/splunk/etc/splunk-launch.conf

0 Karma

somesoni2
Revered Legend

You can go to Splunk UI of the Splunk servers (indexer I believe), go to Settings->Indexes in there, and see the HomePath column to see the actual path were index data is stored. From that path everything before <indexname>/db is the directory where all index data is stored. So, if the path says (example) /opt/splunk/var/lib/splunk/myindex/db (where myindex is the index name), then /opt/splunk/var/lib/splunk/ is your $SPLULNK_DB and you should be copying everything from /opt/splunk/var/lib/splunk/* to new drive. If its /some/other/path/myindex/db, then /some/other/path is your $SPLULNK_DB and you should be copying everything from /some/other/path/* to new drive.

0 Karma
Get Updates on the Splunk Community!

Observability Highlights | November 2022 Newsletter

 November 2022Observability CloudEnd Of Support Extension for SignalFx Smart AgentSplunk is extending the End ...

Avoid Certificate Expiry Issues in Splunk Enterprise with Certificate Assist

This blog post is part 2 of 4 of a series on Splunk Assist. Click the links below to see the other ...

Using Machine Learning for Hunting Security Threats

REGISTER NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more ...