We have Splunk 8.0.3 deployed to a private AWS cloud. We use AWS i3.8xlarge instance types for our indexers, recently upgraded from i3.4xlarge. We combine the 1.7TB "ephemeral" volumes into a logical volume group and use them for splunk index buckets mounted on /opt/splunk/var/lib/splunk. When we were running on i3.4xlarge instances where we had two 1.7 TB volumes, we were using 3 TB of the 3.4 TB logical volume group per indexer as Splunk indexes. When we upgraded to i3.8xlarges we removed the old indexers and the new indexers are only using 200GB of the 6.8TB logical volume groups, slowly creeping up about 4GB/hour. I have tried running searches over long periods of time, but they fail with: ! DAG Execution Exception: Search has been cancelled ! Search auto-canceled ! The search job has failed due to an error. You may be able view the job in the Job Inspector How do I get the cache volumes to fill up again quickly with index data from the S3 storage so my searches will be fast and complete again?
... View more