What is the best way to increase partition containing Splunk?

[mvor]

Our splunk is running on /opt and free space is running low.
My options in adding space are:
1. Extend existing partion
2. Create new partition and rsync /opt to new /opt

Which of these 2 is the better way to go?

[woodcock]

Ideally, you should have the splunk Hot/Warm volume on its own partition/disk, different from Cold (also on its own) and different from the install files (also on its own). You might as well bite the bullet now and fix everything.

[yannK]

Check your indexes.conf,

• if you did customize the path for some indexes or volumes, you can relocate them manually on a per index basis

  • if you did not redefine the path of the indexes, you could change the main variable SPLUNK_DB to relocate your indexes

here is the way for the default location :

• Stop Splunk.

• Mount a new storage, create a special folder for Splunk indexes on it (not not use the main folder for convenience, and make it read/write for the user running Splunk)

• Then move the indexes data from your /opt/splunk/var/lib/splunk to your /newmount/splunkindexes

• Then edit the splunk-launch.conf in $SPLUNK_HOME/etc/ and add SPLUNK_DB=/newmount/splunkindexes

• then start Splunk.

Another alternate solution to keep some data on /opt is to use the coldPath option in indexes.conf to try to ask Splunk to move some buckets to the cold location, and keep the hot and warm on the original location. (but this is not always the best long term solution)

[rabbidroid · Accepted solution]

If it's just your Splunk installation, I don't see why you should not just extend the current partition, which can usually be done live on-the-fly with no downtime, depending on your OS and configuration.

If there are indexes in /opt, I would create a new partition for the indexes, rsync them over there, and linking the new path to the old one. But this has to be done carefully.

[mvor]

The current setup is that both splunk installation and indexes are on one partition (/opt).

[rabbidroid]

That's not the most ideal setup, but if you want to split them up, then the rsync route should be the way to do it, otherwise just increase the current partition.

I highly recommend having the indexes on a separate physical storage than the OS and /opt.

[mvor]

Okay, I'd better fix this setup now rather than later.

What would be the watch-outs on implementing this:
"create a new partition for the indexes, rsync them over there, and linking the new path to the old one"

[rabbidroid]

Make sure permissions are correct on the destination, use rsync -auv /old/index/dir/ /new/index/dir to preserve the timestamps, permission and ownership, when done rename the old index directory with the mv command, then ln -s /new/index/dir /old/index/dir.

Make sure splunk is not running, however you can do an initial rsync while Splunk is up, and when that's sync is done, shut down Splunk, and do the final rsync with the --delete flag. this will delete the files that have disappeared from the index while you were doing the initial sync.

I go for the linking route, because I'm clustered, and I don't want to mess around with indexes.conf by changing the location of the indexes. But if you are not clustered, you can just rename the old folder, create a new folder with the same name, then mount the new partition in the same place the old one was, like this:
mv /index/folder/location /index/folder/location.OLD
mkdir /index/folder/location
mount /index/folder/location
The last one assuming you put it in fstab.
After confirming that everything is in working order, you can delete the old data.

[mvor]

thanks!will implement this one.