Unable to search & getting this error : Unable to evict enough data

[oosegueda]

Hello,

I have been trying to troubleshoot this for a while, and I am not sure if these two things are connected, but, I am not able to search anything in splunk. The UI appears as if nothing was searched, it doesn't even show 0 results.

Also, in the splunkd.log this following error keeps reoccurring:

02-21-2018 01:34:02.312 +0000 INFO  CacheManager - Eviction requested, bytes_needed=7265398784 partition of path=/opt/splunk/var/lib/splunk/audit/db
02-21-2018 01:34:02.312 +0000 WARN  CacheManager - Last run failed to evict requested bytes. Performing eviction in urgent mode for path=/opt/splunk/var/lib/splunk/audit/db
02-21-2018 01:34:02.312 +0000 INFO  CacheManager - Eviction results: count=0, test_count=0, bytes_evicted=0, bytes_needed=7265398784, elapsed_ms=1
02-21-2018 01:34:02.312 +0000 WARN  CacheManager - Unable to evict enough data. Evicted size=0 instead of size=7265398784

[tpeveler_splunk]

A Splunk implementation requires a default minimum free disk space of 5000MB for indexing. In your case it looks like the minimum free disk space limit may have been changed to a higher value. Check out the following two options to address the errors and enable your searches to return results:

1) Add space to the file system that houses Splunk (i.e. /opt/...)

2) Decrease the minimum free disk space limit. See http://docs.splunk.com/Documentation/Splunk/7.0.0/Indexer/Setlimitsondiskusage for details on how to set limits on disk usage.

I had a customer that was experiencing the same issue as yours and we were able to address it by implementing option 1 (add space).