We have Splunk 8.0.3 deployed to a private AWS cloud. We use AWS i3.8xlarge instance types for our indexers, recently upgraded from i3.4xlarge. We combine the 1.7TB "ephemeral" volumes into a logical volume group and use them for splunk index buckets mounted on /opt/splunk/var/lib/splunk. When we were running on i3.4xlarge instances where we had two 1.7 TB volumes, we were using 3 TB of the 3.4 TB logical volume group per indexer as Splunk  indexes. When we upgraded to i3.8xlarges we removed the old indexers and the new indexers are only using 200GB of the 6.8TB logical volume groups, slowly creeping up about 4GB/hour. I have tried running searches over long periods of time, but they fail with: ! DAG Execution Exception: Search has been cancelled ! Search auto-canceled ! The search job has failed due to an error.  You may be able view the job in the Job Inspector How do I get the cache volumes to fill up again quickly with index data from the S3 storage so my searches will be fast and complete again?   ... View more

When I run the "aws" command as a normal user or root, it works. When I run the "aws" command as user splunk, it produces the python error: "ImportError: No module named httpsession". I believe that this is because the version of python in ~splunk/bin is different than the system's version. How do I get the httpsession module installed to Splunk's version of python? ... View more

Answers that have search strings or other text containing the underscore character don't display the underscore.  Instead, they start italicizing text until they see another underscore.  This is going to cause a lot of confusion. ... View more