Isn't it what a multi-site smartstore should have, a single smartstore shared by the indexers from two sites that are part of the same cluster?   Also if they are part of the same cluster, wouldn't Cluster Manager push the indexes.conf across all sites which means, the remote volume details has to be the same across the indexers on each site (like the endpoint, bucket name, credentials)? ... View more

What you wanted to do is bootstrapping. https://docs.splunk.com/Documentation/Splunk/9.0.0/Indexer/BootstrapaSmartStoreindexercluster For the actual command, you can check this thread. https://community.splunk.com/t5/Knowledge-Management/SmartStore-What-is-Cluster-bootstrap-process/m-p/401827 ... View more

I am assuming you meant setting this parameter to TRUE so only one single copy can be maintained in the cache. I don't know the exact details of the metadata maintained, but certainly know the cluster master maintains details about bucket locality (remote vs local, whether a copy is uploaded to remote storage etc).  Given the SmartStore architecture, the indexers are generally stateless so they shouldn't be having metadata info in them.  Note: this is my assumption and might be wrong.  If anyone else has good knowledge about metadata with Splunk SmartStore, please feel free to update. ... View more

This is based on the following settings in server.conf of the indexers. evict_on_stable By default, this parameter is set to false, which means the RF copies are preserved in the cache until they are evicted by the cache eviction policies. ... View more

What about the TA that was developed earlier which supports both 8.1 and < 8.1? ... View more

How about the indexes that were migrated to SmartStore? The homePath remains the same after migration and it refers to a volume that had maxVolumeDataSizeMB. ... View more

This is completely odd. I just migrated some classic indexes that had data in ColdPath and even tried evicting all the buckets, ran a search and it brought the data back to ColdPath. ... View more

Thank you. ... View more

The Cache sizing should be updated given Splunk 8.0+ will have RF number of copies across the cache until they are evicted. When SmartStore was introduced in 7.2, the behavior was to keep only one copy of the bucket in the cache after the warm bucket is uploaded into S3 but would have had RF-1 number of stubs (just metadata). The behavior has changed since 8.0 (not sure if it was from 7.3+) for performance reasons. Global Cache sizing = Daily Ingest Rate x Compression Ratio x RF x Cached Days ... View more

Does Splunk support multiple S3 object stores configured within the same indexer cluster? I understand the indexes.conf certainly allows this and I can configure each index to point to specific S3 storage but wanted to get clarified if this is indeed supported by Splunk. ... View more

This is not true. Warm still exists and Cache is nothing but the total space across Hot & Warm. ... View more

What happens when the versioning is not enabled on the S3 bucket? How does remote.s3.supports_versioning behave? ... View more

It can happen when the target peers didn't receive an acknowledgment from the primary after its upload within the remote_storage_upload_timeout (default is 60 seconds) and the target peer will now check S3 for the receipt.json and if it couldn't find, it will upload its copy to the S3 with its receipt.json. This is a fallback mechanism in case if the primary server goes down for any reason, Splunk still ensures that a copy of the bucket is uploaded by another peer. ... View more

I am seeing the behavior with classic to SmartStore migration in a multisite setup on 8.0, where every indexer that has a copy uploads to S3 and since each has a different uploader_guid, they don't get overwritten except for the receipt.json file which will refer to the last one to be uploaded. I am now testing the same on a single site just to make sure if this is an issue in a multi-site setup. FYI, I had site RF as 3 (origin:2,total:3) and I had 3 copies of the bucket in S3 meaning the two copies from the origin site also got uploaded which is weird. Clearly there seems to be different code path for migration vs standard upload as part of ingest. ... View more

Yes. If you do not update coldPath, any searches that involve data from what used to be the cold tier would still download data from S3 into the coldPath location. After migration to SmartStore, coldPath can certainly be changed but it cannot be same as homePath but the volume can be same except coldPath will have the colddb instead of db. Here is an example. [apache] remotePath = volume:remote_store/$_index_name homePath = volume:hot/apache/db coldPath = volume:hot/apache/colddb ... View more