since two days my splunk application is crashing 5 minutes after starting the application.
In the Splunkd.log I get the following error description:
03-27-2020 13:00:16.418 +0100 ERROR StreamGroup - failed to drain remainder total_sz=59 bytes_freed=25106 avg_bytes_per_iv=425 sth=0x7fcb703feca0: [1585310416, /opt/splunk/var/lib/splunk/audit/db/hot_v1_27, 0x7fcb7570c650] reason=st_sync failed rc=-6 warm_rc=[-4,28]
03-27-2020 13:00:16.434 +0100 ERROR StreamGroup - failed to add corrupt marker to dir=/opt/splunk/var/lib/splunk/audit/db/hot_v1_27 errno=No space left on device
03-27-2020 13:00:16.454 +0100 ERROR BTreeCP - addUpdate: IOException caught: BTree::Exception: Record::writeLE failure in Node::_leafAddUpdate node offset: 24 order: 255 keys:
03-27-2020 13:00:16.466 +0100 ERROR BTreeCP - failed: failed to mkdir /opt/splunk/var/lib/splunk/fishbucket/splunk_private_db/corrupt: No space left on device
03-27-2020 13:00:16.466 +0100 ERROR IndexWriter - failed to add corrupt marker to path='/opt/splunk/var/lib/splunk/audit/db/hot_v1_27' (No space left on device)
I checked the space on the disk and the disk areas are only 28 % in use. So there should be enough space on the disk.
Does anyone have an ideas what is the issue of this problem.