I would like to move the entire index database from "/opt/splunk/var/lib/splunk" to "/opt/splunk/var/lib/splunkdb" which is a new mount point.
I followed the direction from the documentation except I used rsync instead of cp.
It seems that everything works except when I remove the "/opt/splunk/var/lib/splunk" directory (the old index database), and restart splunk, it will add the "/opt/splunk/var/lib/splunk" directory back again. And that directory ("/opt/splunk/var/lib/splunk") contains .dat files and persistentstorage.
I would like to permanently remove the directory and only use the new mount point, "/opt/splunk/var/lib/splunkdb".
Would anyone please help me why splunk keeps adding the old directory back again, and what I can do to prevent this to happen again so that I can only use the new mount point.
You cab change path to indexes in Settings>>System settings » General settings
So that future indexed data will be stored to new location.
Splunk original directory structure remains same. It won't harm you
I confirmed that the path to indexes is correctly configured as "/opt/splunk/var/lib/splunkdb" which is the new mount point and new data is indexed there. The problem is that I cannot figure out why splunk keeps generating .dat files and persistentstorage in the old directory (splunk)although SPLUNK_DB is now pointing to the new directory(splunkdb).
As a precaution, make sure all indexes.conf stanzas actually do use
$SPLUNK_DB and not an absolute fixed path to the old location.
Check your configurations for that index via btool-
splunk btool indexes list myindexname --debug
That will show you all configurations applied to that index. You might have some left over configuration in there.