How do I get SmartStore to fill its cache again?

esalesap — Tue, 09 Nov 2021 05:28:19 GMT

We have Splunk 8.0.3 deployed to a private AWS cloud.

We use AWS i3.8xlarge instance types for our indexers, recently upgraded from i3.4xlarge.

We combine the 1.7TB "ephemeral" volumes into a logical volume group and use them for splunk index buckets mounted on /opt/splunk/var/lib/splunk.

When we were running on i3.4xlarge instances where we had two 1.7 TB volumes, we were using 3 TB of the 3.4 TB logical volume group per indexer as Splunk indexes.

When we upgraded to i3.8xlarges we removed the old indexers and the new indexers are only using 200GB of the 6.8TB logical volume groups, slowly creeping up about 4GB/hour.

I have tried running searches over long periods of time, but they fail with:

! DAG Execution Exception: Search has been cancelled
! Search auto-canceled
! The search job has failed due to an error. You may be able view the job in the Job Inspector

How do I get the cache volumes to fill up again quickly with index data from the S3 storage so my searches will be fast and complete again?

Re: How do I get SmartStore to fill its cache again?

esalesap — Wed, 10 Nov 2021 20:36:23 GMT

Ok, so the "DAG Execution" errors were caused by me running long-running searches in multiple browser tabs. The errors would occur if I switched between tabs. Running searches in their own windows solved the search error problem.

I'm still looking for a fast way to stimulate the indexers to load previously indexed data from S3 to the indexers.

topic How do I get SmartStore to fill its cache again? in Splunk Search

How do I get SmartStore to fill its cache again?

Re: How do I get SmartStore to fill its cache again?