Error while searching and other error in the notification message?

[pacifikn]

Dear Team,

Greetings!!

I need your help and guidance on the following issue , i keep getting this error in the notifications message:

Search peer Splunk-idx4 has the following message: The minimum dree disk space (5000MB)
reached for opt/splunk/var/run/splunk/dispatch.

Problem replicating config (bundle) to search peer '10.10.5.106:8089' , HTTP response code 500
(HTTP/1.1 500 Error writing to /opt/splunk/var/run/searchpeers/Splunksh01-1642302054.bundle.53d7c4e2bfaedd1d.tmp:
NO space left on device).
Error writing to /opt/splunk/var/run/searchpeers/Splunksh01-1642302054.bundle.fbc779696ccbf76a.tmp:
No space left on the device (unknown write error)

Even on the search and reporting when i run a query, it gives this error, 

2 error occurred while the search was executing. Therefore, search results might be incomplete. Hide errors

. [Splunk id-03] Failed to read size=3307 event(s) from raw data in bucket='nsoc_fw_ahnlab~703~B239BEEE-90FA-43C8-ADDA-620D3FACAB66' path ='/opt/splunk_data/indexes/nsoc_fw_ahnlab/db/hot_v1_703. Rawdata may be corrupt, see seach log. Results may be incomplete!

. [Splunk id-03] Failed to read size=5030 event(s) from raw data in bucket='nsoc_fw_ahnlab~703~B239BEEE-90FA-43C8-ADDA-620D3FACAB66' path ='/opt/splunk_data/indexes/nsoc_fw_ahnlab/db/hot_v1_703. Rawdata may be corrupt, see seach log. Results may be incomplete!

Kindly help me and guide me on how to fix the above issue.

Thank you in advance!

[richgalloway]

You have what appear to be two different problems.

The first problem is a lack of available disk space on indexer Splunk-idx4.  Check /opt/splunk/var/run/searchpeers for old bundle files and delete them to free up some space.

Consider adding more space to the volume.  If it isn't already, make the storage for /opt/splunk distinct from other volumes, especially root.

The second problem is possible corrupt index buckets.  See the search logs mentioned in the notifications for details.  You'll probably need to run Splunk's fsck utility to find and repair the problem.