Monitoring Splunk

Move Splunk's VAR folder ($SPLUNK_HOME/var or /opt/splunk/var)

BP9906
Builder

I've had Splunk for sever major releases (4.x to present) and now as our environment has evolved, our /opt/splunk/var/* path has many disk writes. Since Splunk v4, I've always moved SPLUNK_DB (ie /opt/splunk/var/lib/splunk) to a separate partition for indexers so that I can dedicate my fast disks (RAID 10) to indexing appropriately.

Now, I see many writes on /opt/splunk/var (yet my SPLUNK_DB resides elsewhere). I've got indexer clustering on some servers and search head clustering on others, and they all show the same behavior. I suspect its bundle replication /opt/splunk/var/run/ because I get an occasional warning that the configuration initialization took a little longer than normal.

How can I get the disk writes out of /opt/splunk/var and into my other drive with raid10 without having to move the entire splunk home folder?

0 Karma
1 Solution

woodcock
Esteemed Legend

The simplest way is to shut down Splunk, create a new directory on your other drive whereever you like and named whatever you like, such as mkdir /mnt/otherdrive/OptSplunkVarRun, move all the files with mv /opt/splunk/var/run/* /mnt/otherdrive/OptSplunkVarRun/, remove the old directory with rmdir /opt/splunk/var/run then create a soft link with ln -fs /mnt/otherdrive/OptSplunkVarRun /opt/splunk/var/run, and finally restart Splunk. I had to do this with the dispatch directory and it worked fine.

View solution in original post

harry2007gsp
Path Finder

Hi @woodcock, I tried your method and it created the soft link from external network drive to this directory /opt/splunk/val

But I can see data is still stored on the local storage rather than on network storage.

0 Karma

woodcock
Esteemed Legend

Follow all the steps. Make sure that you do each one exactly. If you think that it didn't work, post the output of df.

0 Karma

woodcock
Esteemed Legend

The simplest way is to shut down Splunk, create a new directory on your other drive whereever you like and named whatever you like, such as mkdir /mnt/otherdrive/OptSplunkVarRun, move all the files with mv /opt/splunk/var/run/* /mnt/otherdrive/OptSplunkVarRun/, remove the old directory with rmdir /opt/splunk/var/run then create a soft link with ln -fs /mnt/otherdrive/OptSplunkVarRun /opt/splunk/var/run, and finally restart Splunk. I had to do this with the dispatch directory and it worked fine.

View solution in original post

Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.