Indexers Disk Space

[riqbal47010]

all of our indexers server disk space is almost 90% full and one of the indexer server disk is full(100%) so he get stopped.

So the first thing is to determine that why only specific server disk space get full(100%) and others are at 90% .

secondly how can we solve this issue.

[diogofgm]

Whats is your setup? do you have indexer clustering? Can you post you indexers indexes.conf?

Its always possible to have different percentages of usage since the sources (e.g. forwarders) might not be sending data equally to both indexers. Splunk UFs can do Load balancing, but it usually timebased so there is still a chance that you can get more data in one interval and less on the other. Size based LB is available on more recent versions.

You should be using volume management and better retention policies to avoid having having you indexers to stop due to lack of space.

Check this link:
https://docs.splunk.com/Documentation/Splunk/7.3.1/Indexer/Configureindexstoragesize

[riqbal47010]

thanks for the reply, interestingly after removing excessive buckets I got enough disk space.
we have nine months retention policy after that they moved to Frozen NFS share.

thanks for reply.

[diogofgm]

great! if my answers help you please make it as such.
Still be aware that bigger issues might arise if you don't design your infra and storage policies properly since, depending on your cluster search factor and replication factor, if one of your indexers goes down the other will eventually start fixup tasks building the missing buckets and filling up your storage.

[riqbal47010]

hi diogofgm,

we have different indexes.conf for each technology index. Now the two indexers have 96% utilized disk space remaining others are 75 to 80 percent utilized. how can we immediately solve this issue.

Further to that, If i reduce the retention period to 6 months from 9 months. will it fix the disk space issue. primarily we need 6 months data online.

[diogofgm]

yes, if you reduce the retention period it will likely fix your disk space issue. but bear in mind that a bucket will only roll/be deleted when all the data it contains hits the retention period. Meaning buckets with several days worth of data will only roll / be deleted when after all the data has passed the retention period.
Also, difference in usage its possible depending on how you're sending data and if you have Splunk LB setup. check this: https://docs.splunk.com/Documentation/Splunk/7.3.1/Forwarding/Setuploadbalancingd