How to re-register smartstore s3 bucket contents with cache manager?

[Glasses]

Hi,

We were doing some testing in a lab environment, with a s2 idx cluster.   Apparently a lab user shutdown/rebooted the wrong nodes, resulting in the cluster master and one or more of the 3 cluster members were shutdown by accident in the lab.

After starting the idx cluster boxes and restarting splunk, we have noticed some discrepancies in the data we are searching, e.g. we are not seeing all the data we used to see.  There appear to be gaps.

We believe the cache manager was corrupted as the data seems to be complete in the s2 (s3 bucket) but not when searching.

We are looking the proper way to reconnect to the existing smartstore (s3 bucket path) and reset the cache manager to re-register all the contents again.

Unfortunately I am not finding the documentation, which I thought I read months ago...

Related to this, one of our use-cases was to test connecting a standalone indexer to the same s2 to retrieve previous years of logs for forensic reasons, without impacting the primary s2 idx cluster.

Thank you in advance 

[srajarat2]

What you wanted to do is bootstrapping.

https://docs.splunk.com/Documentation/Splunk/9.0.0/Indexer/BootstrapaSmartStoreindexercluster

For the actual command, you can check this thread.

https://community.splunk.com/t5/Knowledge-Management/SmartStore-What-is-Cluster-bootstrap-process/m-...

[scelikok]

@Glasses, maybe the local index files are corrupted. You can use the below rest call on each indexer to evict all buckets. When you search they will be downloaded again from S3.

curl -ku admin:changeme "https://localhost:8089/services/admin/cacheman/_evict" -d path=/path_to_localcache/ -d mb=99999999999

[amehta_splunk]

I'm hoping the parent message solved your issue.  If you did, there's another Splunk Answer with more detail on:

• services/admin/cacheman/_evict
• services/admin/cacheman/_localize

That thread might provide some necessary context on how to use the commands - and when.

https://community.splunk.com/t5/Deployment-Architecture/How-to-evict-buckets-from-smartstore-cache/m...

[Glasses]

Thank you for the reply.

I will take try your suggestion, of evicting the local data, and seeing if we can pull back again from s3.

Other than evict is there anything special to do to get the cache manager to read the s3 smartstore bucket?

Do we need to do a rolling restart or delete any files?

Does the cache manager automatically detect anything in the smartstore s3 bucket?

Thank you again!

[scelikok · Accepted solution]

You don’t need to do anything else. Indexers will download needed files from S3 when you search. 
Rolling Restart will also do the same since after restart indexers will mark all local cache as invalid.