Splunk Enterprise

Disk Space Issue on /opt/splunk


Hi Splunk Folks,

We have Splunk Physical Servers with 8GB disk space storage for /opt folder which frequently reaching 90% of the disk space threshold (7.2GB). Since we cannot easily upgrade the disk space because these are Physical servers, we are looking for files that we can remove or migrate.

We found this "/opt/splunk/var/lib/splunk/fishbucket/splunk_private_db/save" folder (1GB in size) that seems like containing the same files (btree_index.dat, btree_records.dat and snapshot) with its predecessor folder (/opt/splunk/var/lib/splunk/fishbucket/splunk_private_db)

Are questions are, what are these Splunk files do and does it safe if we will delete or move them to another folder to free some disk space on /opt?

Here is the commands we used to check which file has consume a large volume of diskspace

-bash-4.2$ df -h /opt/splunk
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg00-opt 8.0G 6.5G 1.6G 82% /opt

-bash-4.2$ du -h --max-depth=1 /opt/splunk/var/lib/splunk/fishbucket/splunk_private_db
1001M /opt/splunk/var/lib/splunk/fishbucket/splunk_private_db/save (Has the most consumed diskspace)
335M /opt/splunk/var/lib/splunk/fishbucket/splunk_private_db/snapshot
1.7G /opt/splunk/var/lib/splunk/fishbucket/splunk_private_db/ (Total)

If we look inside the "save" folder from /opt/splunk/var/lib/splunk/fishbucket/splunk_private_db, we can see it has same files (btree_index.dat, btree_records.dat and snapshot) . Thus it just might be a backup of splunk_private_db

-bash-4.2$ ls -l /opt/splunk/var/lib/splunk/fishbucket/splunk_private_db
-rw-------. 1 splunk splunk 104865400 Jun 24 04:52 btree_index.dat
-rw-------. 1 splunk splunk 246211800 Jun 24 04:56 btree_records.dat
drwx------. 3 splunk splunk 79 Jun 24 04:49 save
drwx------. 2 splunk splunk 70 Jun 24 04:49 snapshot

-bash-4.2$ ls -l /opt/splunk/var/lib/splunk/fishbucket/splunk_private_db/save
-rw-------. 1 splunk splunk 152715440 Nov 22 2019 btree_index.dat
-rw-------. 1 splunk splunk 371572840 Nov 22 2019 btree_records.dat
drwx------. 2 splunk splunk 70 Nov 22 2019 snapshot



John Kevin Aracan

Labels (2)
Tags (1)
0 Karma


Based on timestamps it’s probably your backup dir for some reason. I suppose that you could remove or move those to some other place.

0 Karma
Get Updates on the Splunk Community!

Splunk Lantern | Getting Started with Edge Processor, Machine Learning Toolkit ...

Splunk Lantern is Splunk’s customer success center that provides advice from Splunk experts on valuable data ...

Enterprise Security Content Update (ESCU) | New Releases

In the last month, the Splunk Threat Research Team (STRT) has had 2 releases of new security content via the ...

Announcing the 1st Round Champion’s Tribute Winners of the Great Resilience Quest

We are happy to announce the 20 lucky questers who are selected to be the first round of Champion's Tribute ...