I've read some Answers on this issue and understand how to solve by adjusting server.conf. The question i have is how exactly to trace back this error to the object (search, report, alerts, etc...) that causing this issue. We have couple thousand of these objects and multiple search clusters and 20 indexes in the cluster. It would be great to have steps to isolate the offending object.
Hi @brdr ,
This message is usually an indication of improperly configured storage for indexing operations. If you're running into a situation where your indexers have less than 5GB (the default threshold for this message) of free disk space for the hot/cold storage volumes, you probably have one of the following situations:
1. You have not properly configured indexes.conf settings for volume management that allows Splunk to clean up space as needed for the hot/cold volumes.
2. You have not provided the supported default minimum disk space for /opt/splunk (or wherever Splunk is installed) of 300GB and search operations are overfilling that space causing this message.
From your desription, it sounds like #2 in this case. If it were me, I would investigate the server(s) in question to determine which folder is causing the issue (most likely something in /opt/splunk/var). From the CLI (Linux) I would use the following command:
df -sh /opt/splunk
df -sh /opt/splunk/var
If you have NOT allocated the default minimum of 300GB for /opt/splunk, I would highly recommend that you do that. If the hot/cold data shares the same mount point as /opt/splunk then I would recommend that you review your indexes.conf to implement volume/index management that does not allow your disk to fill up, and rolls data appropriately to do this. Typically, I recommend that configurations be made to leave a 5% - 10% chunk of free space on the volume.
I hope this helps.
I got into similar issue.
My indexers were having huge files under $SPLUNK_HOME/var/run/searchpeers.
Which are images of all unwanted lookup files created on the search heads. After clearing them on search heads, the disk space came down on indexers.
Did you search for such files? or done some housekeeping?