Monitoring Splunk

Move Splunk's VAR folder ($SPLUNK_HOME/var or /opt/splunk/var)

BP9906
Builder

I've had Splunk for sever major releases (4.x to present) and now as our environment has evolved, our /opt/splunk/var/* path has many disk writes. Since Splunk v4, I've always moved SPLUNK_DB (ie /opt/splunk/var/lib/splunk) to a separate partition for indexers so that I can dedicate my fast disks (RAID 10) to indexing appropriately.

Now, I see many writes on /opt/splunk/var (yet my SPLUNK_DB resides elsewhere). I've got indexer clustering on some servers and search head clustering on others, and they all show the same behavior. I suspect its bundle replication /opt/splunk/var/run/ because I get an occasional warning that the configuration initialization took a little longer than normal.

How can I get the disk writes out of /opt/splunk/var and into my other drive with raid10 without having to move the entire splunk home folder?

0 Karma
1 Solution

woodcock
Esteemed Legend

The simplest way is to shut down Splunk, create a new directory on your other drive whereever you like and named whatever you like, such as mkdir /mnt/otherdrive/OptSplunkVarRun, move all the files with mv /opt/splunk/var/run/* /mnt/otherdrive/OptSplunkVarRun/, remove the old directory with rmdir /opt/splunk/var/run then create a soft link with ln -fs /mnt/otherdrive/OptSplunkVarRun /opt/splunk/var/run, and finally restart Splunk. I had to do this with the dispatch directory and it worked fine.

View solution in original post

harry2007gsp
Path Finder

Hi @woodcock, I tried your method and it created the soft link from external network drive to this directory /opt/splunk/val

But I can see data is still stored on the local storage rather than on network storage.

0 Karma

woodcock
Esteemed Legend

Follow all the steps. Make sure that you do each one exactly. If you think that it didn't work, post the output of df.

0 Karma

woodcock
Esteemed Legend

The simplest way is to shut down Splunk, create a new directory on your other drive whereever you like and named whatever you like, such as mkdir /mnt/otherdrive/OptSplunkVarRun, move all the files with mv /opt/splunk/var/run/* /mnt/otherdrive/OptSplunkVarRun/, remove the old directory with rmdir /opt/splunk/var/run then create a soft link with ln -fs /mnt/otherdrive/OptSplunkVarRun /opt/splunk/var/run, and finally restart Splunk. I had to do this with the dispatch directory and it worked fine.

Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...