Error of Problem replicating config (bundle) to search peer ' 10.10.x.79:8089 ',

[pacifikn]

Greetings!!

I need help!!! am experiencing an error while am doing search, the error is:

Search peer Splkidx04 has the following
message: The minimum free disk space
(5000MB) reached for /opt/splunk/var/run/splunk/dispatch.

Problem replicating config (bundle) to search peer ' 10.10.x.96:8089 ',
HTTP response code 500 (HTTP/1.1 500 Error writing to
/opt/splunk/var/run/searchpeers/Splunksh1-1641956462.bundle.4ef204fd344a6181.tmp:
No space left on device). Error writing to
/opt/splunk/var/run/searchpeers/Splunksh01-1641956462.bundle.4ef204fd344a6181.tmp:
No space left on device (Unknown write error) .
1/12/2022, 10:44:22 AM

The search process with sid=scheduler__pacyn__search__RMD5837e19b530431259_at_1641973200_94478
on peer=Spkidx4 might have returned partial results due to a reading error while waiting for the peer.
This can occur if the peer unexpectedly closes or resets the connection during a planned restart.
Try running the search again. Learn more.
1/12/2022, 9:43:09 AM

Search peer Splkidx4 has the following message: The minimum free disk space (5000MB)
reached for /opt/splunk/var/run/splunk/dispatch.
1/12/2022, 10:59:00 AM

5 errors has occurred while the search was executing. therefore search results might be incomplete. hide .....

Kindly help me on how i can fix this above issue. 

Thank you in advance!

[isoutamo]

Hi

one of your indexer is running out of free space (less than 5Gb left).  You should add more space there adding more or removing something.

Of you are using LVM on linux then easiest way it just add additional disk to volume group and then extend splunk partition.

The best way is start to using LVM (if you didn't  use it yet) and also Splunk internal volumes. How this can do in your environment probably needs some work after properly planning it first.

r. Ismo

[pacifikn]

Thank you so muchMr @isoutamo  for your help, yes we actually use LVM,

And as i trace all the indexers i found in :

/dev/mapper/centos-opt , on some of indexer is almost full 92%, and on the other indexer is full 100%, 

what are the way forward to resolve this, May you please guide me on how to fix this? there is no way to reduce this by delete maybe the old data instead of add other space? need your advice and guidance on this,

Thank you in advance!

[isoutamo]

If you have some thawed data you could move it away from /opt or maybe there are some old logs or other stuff what you could move (even temporary) away to get that 5Gb free. The best option is move all indexes to own splunk dedicated partition and then use splunk volumes to restrict it’s space as needed.