Knowledge Management

How to re-register smartstore s3 bucket contents with cache manager?

Glasses
Contributor

Hi,

We were doing some testing in a lab environment, with a s2 idx cluster.   Apparently a lab user shutdown/rebooted the wrong nodes, resulting in the cluster master and one or more of the 3 cluster members were shutdown by accident in the lab.

After starting the idx cluster boxes and restarting splunk, we have noticed some discrepancies in the data we are searching, e.g. we are not seeing all the data we used to see.  There appear to be gaps.

We believe the cache manager was corrupted as the data seems to be complete in the s2 (s3 bucket) but not when searching.

We are looking the proper way to reconnect to the existing smartstore (s3 bucket path) and reset the cache manager to re-register all the contents again.

Unfortunately I am not finding the documentation, which I thought I read months ago...

Related to this, one of our use-cases was to test connecting a standalone indexer to the same s2 to retrieve previous years of logs for forensic reasons, without impacting the primary s2 idx cluster.

Thank you in advance 

Labels (2)
Tags (2)
0 Karma

scelikok
Champion

@Glasses, maybe the local index files are corrupted. You can use the below rest call on each indexer to evict all buckets. When you search they will be downloaded again from S3.

curl -ku admin:changeme "https://localhost:8089/services/admin/cacheman/_evict" -d path=/path_to_localcache/ -d mb=99999999999

  

If this reply helps you an upvote is appreciated.
0 Karma

Glasses
Contributor

Thank you for the reply.

I will take try your suggestion, of evicting the local data, and seeing if we can pull back again from s3.

Other than evict is there anything special to do to get the cache manager to read the s3 smartstore bucket?

Do we need to do a rolling restart or delete any files?

Does the cache manager automatically detect anything in the smartstore s3 bucket?

Thank you again!

0 Karma

scelikok
Champion

You don’t need to do anything else. Indexers will download needed files from S3 when you search. 
Rolling Restart will also do the same since after restart indexers will mark all local cache as invalid. 

If this reply helps you an upvote is appreciated.
0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.