Isn't it what a multi-site smartstore should have, a single smartstore shared by the indexers from two sites that are part of the same cluster?   Also if they are part of the same cluster, wouldn't Cluster Manager push the indexes.conf across all sites which means, the remote volume details has to be the same across the indexers on each site (like the endpoint, bucket name, credentials)? ... View more

What you wanted to do is bootstrapping. https://docs.splunk.com/Documentation/Splunk/9.0.0/Indexer/BootstrapaSmartStoreindexercluster For the actual command, you can check this thread. https://community.splunk.com/t5/Knowledge-Management/SmartStore-What-is-Cluster-bootstrap-process/m-p/401827 ... View more

I am assuming you meant setting this parameter to TRUE so only one single copy can be maintained in the cache. I don't know the exact details of the metadata maintained, but certainly know the cluster master maintains details about bucket locality (remote vs local, whether a copy is uploaded to remote storage etc).  Given the SmartStore architecture, the indexers are generally stateless so they shouldn't be having metadata info in them.  Note: this is my assumption and might be wrong.  If anyone else has good knowledge about metadata with Splunk SmartStore, please feel free to update. ... View more

This is based on the following settings in server.conf of the indexers. evict_on_stable By default, this parameter is set to false, which means the RF copies are preserved in the cache until they are evicted by the cache eviction policies. ... View more

What about the TA that was developed earlier which supports both 8.1 and < 8.1? ... View more

How about the indexes that were migrated to SmartStore? The homePath remains the same after migration and it refers to a volume that had maxVolumeDataSizeMB. ... View more

This is completely odd. I just migrated some classic indexes that had data in ColdPath and even tried evicting all the buckets, ran a search and it brought the data back to ColdPath. ... View more

Thank you. ... View more

The Cache sizing should be updated given Splunk 8.0+ will have RF number of copies across the cache until they are evicted. When SmartStore was introduced in 7.2, the behavior was to keep only one copy of the bucket in the cache after the warm bucket is uploaded into S3 but would have had RF-1 number of stubs (just metadata). The behavior has changed since 8.0 (not sure if it was from 7.3+) for performance reasons. Global Cache sizing = Daily Ingest Rate x Compression Ratio x RF x Cached Days ... View more

Does Splunk support multiple S3 object stores configured within the same indexer cluster? I understand the indexes.conf certainly allows this and I can configure each index to point to specific S3 storage but wanted to get clarified if this is indeed supported by Splunk. ... View more

This is not true. Warm still exists and Cache is nothing but the total space across Hot & Warm. ... View more

What happens when the versioning is not enabled on the S3 bucket? How does remote.s3.supports_versioning behave? ... View more

It can happen when the target peers didn't receive an acknowledgment from the primary after its upload within the remote_storage_upload_timeout (default is 60 seconds) and the target peer will now check S3 for the receipt.json and if it couldn't find, it will upload its copy to the S3 with its receipt.json. This is a fallback mechanism in case if the primary server goes down for any reason, Splunk still ensures that a copy of the bucket is uploaded by another peer. ... View more

I am seeing the behavior with classic to SmartStore migration in a multisite setup on 8.0, where every indexer that has a copy uploads to S3 and since each has a different uploader_guid, they don't get overwritten except for the receipt.json file which will refer to the last one to be uploaded. I am now testing the same on a single site just to make sure if this is an issue in a multi-site setup. FYI, I had site RF as 3 (origin:2,total:3) and I had 3 copies of the bucket in S3 meaning the two copies from the origin site also got uploaded which is weird. Clearly there seems to be different code path for migration vs standard upload as part of ingest. ... View more

Yes. If you do not update coldPath, any searches that involve data from what used to be the cold tier would still download data from S3 into the coldPath location. After migration to SmartStore, coldPath can certainly be changed but it cannot be same as homePath but the volume can be same except coldPath will have the colddb instead of db. Here is an example. [apache] remotePath = volume:remote_store/$_index_name homePath = volume:hot/apache/db coldPath = volume:hot/apache/colddb ... View more

I haven't seen this behavior yet. I am assuming you are testing in an indexer cluster environment. I will be testing some more this weekend. I will update. ... View more

Isn't this opposite of what versioning is supposed to do? As per Splunk's indexes.conf documentation from https://docs.splunk.com/Documentation/Splunk/8.0.0/Admin/Indexesconf it clearly says versioning is a means of keeping multiple variants of an object. remote.s3.supports_versioning = <boolean> * Specifies whether the remote storage supports versioning. * Versioning is a means of keeping multiple variants of an object in the same bucket on the remote storage. * Optional. * Default: true Hence I would expect a delete marker in place when an object is deleted. Can you clarify? ... View more

The local data is not evicted by default after the migration. You can validate them by checking by running |rest /services/admin/cacheman | dedup title and verify the buckets, its status, size etc., ... View more

Thank you very much. I have been looking out for this exact info for a while. ... View more

Based on our troubleshooting, Splunk is issuing a get_object of receipt.json (with the whole bucket details in the object name) before even performing a put_object and this should result in 404. As per Splunk support this can be ignored unless you see 404 for the same object multiple times. ... View more

Splunk Support did confirm that SmartStore is only supported on Linux at this time and not on Windows. ... View more

Alternatively, you can set up a new bucket for the second environment on the first S3 object store, replicate the data from the second S3 object store to first (assuming they are on similar object stores with replication enabled), include the new bucket under a new volume in indexes.conf. ... View more

I am assuming you are including "dedup title" on the search with "rest /services/admin/cacheman" as S3 keeps only a single copy of the buckets. Meanwhile, the cm:bucket.estimated_size is the actual bucket size and there is no guarantee that the local storage would have all of it as buckets could have been evicted. Alternatively, adding the bucket size across all deduped buckets will give you the total S3 storage use. |rest /services/admin/cacheman | dedup title | stats sum(cm:bucket.estimated_size) AS remoteDiskSize | eval remoteDiskSizeGB = round(remoteDiskSize / 1024 / 1024 / 1024, 2) If you are still seeing the cm:bucket.status as local means there might be some files left out on that bucket like bucket_info.csv or Hosts.data or Sources.data etc., Try evicting the bucket using the following command. splunk _internal call /services/admin/cacheman/"bid|index-name~id~guid|"/evict -method POST -auth un:pwd Replace index-name, id and guid as follows. If the bucket you wanted to evict under an index named apache is db_1572849480_1572840108_19_9A1BE399-F73B-4F0B-83E4-43F7959E3710 invoke as follows. splunk _internal call /services/admin/cacheman/"bid|apache~19~9A1BE399-F73B-4F0B-83E4-43F7959E3710|"/evict -method POST ... View more

Seems like Splunk SmartStore is not supported on Windows, yet? Check this link. https://docs.splunk.com/Documentation/Splunk/8.0.0/Indexer/SmartStoresystemrequirements#Operating_system_requirements It refers to Linux only. ... View more

https://docs.splunk.com/Documentation/Splunk/7.3.2/DistSearch/Forwardsearchheaddata ... View more