Splunk Enterprise

Disk Space Issue on /opt/splunk


Hi Splunk Folks,

We have Splunk Physical Servers with 8GB disk space storage for /opt folder which frequently reaching 90% of the disk space threshold (7.2GB). Since we cannot easily upgrade the disk space because these are Physical servers, we are looking for files that we can remove or migrate.

We found this "/opt/splunk/var/lib/splunk/fishbucket/splunk_private_db/save" folder (1GB in size) that seems like containing the same files (btree_index.dat, btree_records.dat and snapshot) with its predecessor folder (/opt/splunk/var/lib/splunk/fishbucket/splunk_private_db)

Are questions are, what are these Splunk files do and does it safe if we will delete or move them to another folder to free some disk space on /opt?

Here is the commands we used to check which file has consume a large volume of diskspace

-bash-4.2$ df -h /opt/splunk
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg00-opt 8.0G 6.5G 1.6G 82% /opt

-bash-4.2$ du -h --max-depth=1 /opt/splunk/var/lib/splunk/fishbucket/splunk_private_db
1001M /opt/splunk/var/lib/splunk/fishbucket/splunk_private_db/save (Has the most consumed diskspace)
335M /opt/splunk/var/lib/splunk/fishbucket/splunk_private_db/snapshot
1.7G /opt/splunk/var/lib/splunk/fishbucket/splunk_private_db/ (Total)

If we look inside the "save" folder from /opt/splunk/var/lib/splunk/fishbucket/splunk_private_db, we can see it has same files (btree_index.dat, btree_records.dat and snapshot) . Thus it just might be a backup of splunk_private_db

-bash-4.2$ ls -l /opt/splunk/var/lib/splunk/fishbucket/splunk_private_db
-rw-------. 1 splunk splunk 104865400 Jun 24 04:52 btree_index.dat
-rw-------. 1 splunk splunk 246211800 Jun 24 04:56 btree_records.dat
drwx------. 3 splunk splunk 79 Jun 24 04:49 save
drwx------. 2 splunk splunk 70 Jun 24 04:49 snapshot

-bash-4.2$ ls -l /opt/splunk/var/lib/splunk/fishbucket/splunk_private_db/save
-rw-------. 1 splunk splunk 152715440 Nov 22 2019 btree_index.dat
-rw-------. 1 splunk splunk 371572840 Nov 22 2019 btree_records.dat
drwx------. 2 splunk splunk 70 Nov 22 2019 snapshot



John Kevin Aracan

Labels (2)
Tags (1)
0 Karma


Based on timestamps it’s probably your backup dir for some reason. I suppose that you could remove or move those to some other place.

0 Karma
Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!