We have Splunk 8.0.3 deployed to a private AWS cloud. We use AWS i3.8xlarge instance types for our indexers, recently upgraded from i3.4xlarge. We combine the 1.7TB "ephemeral" volumes into a logical volume group and use them for splunk index buckets mounted on /opt/splunk/var/lib/splunk. When we were running on i3.4xlarge instances where we had two 1.7 TB volumes, we were using 3 TB of the 3.4 TB logical volume group per indexer as Splunk  indexes. When we upgraded to i3.8xlarges we removed the old indexers and the new indexers are only using 200GB of the 6.8TB logical volume groups, slowly creeping up about 4GB/hour. I have tried running searches over long periods of time, but they fail with: ! DAG Execution Exception: Search has been cancelled ! Search auto-canceled ! The search job has failed due to an error.  You may be able view the job in the Job Inspector How do I get the cache volumes to fill up again quickly with index data from the S3 storage so my searches will be fast and complete again?   ... View more

When I run the "aws" command as a normal user or root, it works. When I run the "aws" command as user splunk, it produces the python error: "ImportError: No module named httpsession". I believe that this is because the version of python in ~splunk/bin is different than the system's version. How do I get the httpsession module installed to Splunk's version of python? ... View more

Using the following search, I'm seeing AWS CloudTrail ingest lag between 4 and 9 hours. index=ibp_aws sourcetype=aws:cloudtrail* | eval lag=round((_indextime - _time)/60,1) | bin _time span=10m | stats max(lag) AS xLagH min(lag) AS nLagH count by _indextime | eval _time=_indextime | timechart span=10m max(xLagH) min(nLagH) sum(count) If the search is correct, any idea why AWS CloudTrail ingest would lag like this? I'm on Splunk Enterprise 7.0.1 and Splunk_TA_aws 4.4.0. ... View more