I have been trying to troubleshoot this for a while, and I am not sure if these two things are connected, but, I am not able to search anything in splunk. The UI appears as if nothing was searched, it doesn't even show 0 results.
Also, in the splunkd.log this following error keeps reoccurring:
02-21-2018 01:34:02.312 +0000 INFO CacheManager - Eviction requested, bytes_needed=7265398784 partition of path=/opt/splunk/var/lib/splunk/audit/db
02-21-2018 01:34:02.312 +0000 WARN CacheManager - Last run failed to evict requested bytes. Performing eviction in urgent mode for path=/opt/splunk/var/lib/splunk/audit/db
02-21-2018 01:34:02.312 +0000 INFO CacheManager - Eviction results: count=0, test_count=0, bytes_evicted=0, bytes_needed=7265398784, elapsed_ms=1
02-21-2018 01:34:02.312 +0000 WARN CacheManager - Unable to evict enough data. Evicted size=0 instead of size=7265398784
A Splunk implementation requires a default minimum free disk space of 5000MB for indexing. In your case it looks like the minimum free disk space limit may have been changed to a higher value. Check out the following two options to address the errors and enable your searches to return results:
1) Add space to the file system that houses Splunk (i.e. /opt/...)