topic Re: [SmartStore]Can you provide me with some clarification on remote.s3.supports_versioning configuration for SmartStore? in Deployment Architecture

[SmartStore]Can you provide me with some clarification on remote.s3.supports_versioning configuration for SmartStore?

rbal_splunk — Fri, 16 Nov 2018 23:05:07 GMT

When a bucket exceeds the configured data retention time and the parameter remote.s3.supports_versioning=true, then my understanding is that SmartStore will put a delete marker on the corresponding bucket that gets frozen and this data/bucket is ignored by SmartStore for any subsequent searches.

I'm seeing that the bucket gets completely deleted with no delete marker. I wanted to make sure that there's no other configuration that needs to be done other than:

Enable versioning on the S3 bucket
Ensure that remote.s3.supports_versioning=true (default)

Re: [SmartStore]Can you provide me with some clarification on remote.s3.supports_versioning configuration for SmartStore?

rbal_splunk — Tue, 29 Sep 2020 22:00:33 GMT

When versioning is enabled on the S3 bucket:

If s3.supports_versioning=true, then the bucket/data is deleted w/o delete markers.
If s3.supports_versioning=false, then data is deleted and the bucket/data contains delete markers

Re: [SmartStore]Can you provide me with some clarification on remote.s3.supports_versioning configuration for SmartStore?

srajarat2 — Fri, 06 Dec 2019 02:02:29 GMT

Isn't this opposite of what versioning is supposed to do?

As per Splunk's indexes.conf documentation from https://docs.splunk.com/Documentation/Splunk/8.0.0/Admin/Indexesconf it clearly says versioning is a means of keeping multiple variants of an object.

remote.s3.supports_versioning = <boolean>
* Specifies whether the remote storage supports versioning.
* Versioning is a means of keeping multiple variants of an object
  in the same bucket on the remote storage.
* Optional.
* Default: true

Hence I would expect a delete marker in place when an object is deleted. Can you clarify?

Re: [SmartStore]Can you provide me with some clarification on remote.s3.supports_versioning configuration for SmartStore?

srajarat2 — Thu, 12 Dec 2019 21:12:34 GMT

What happens when the versioning is not enabled on the S3 bucket? How does remote.s3.supports_versioning behave?

Re: [SmartStore]Can you provide me with some clarification on remote.s3.supports_versioning configuration for SmartStore?

rbal_splunk — Fri, 13 Dec 2019 00:12:05 GMT

Just to complete this discussion.

When remote.s3.supports_versioning = true , we iterate over all versions of an S3 object (file) and remove all versions. Otherwise, we do a simple remove on the object. This means that if set to true, all versions will be removed and the object contents are irretrievable.

If set to false, the behavior is as follows:
1) if bucket versioning is disabled, the object is simply gone forever;
2) if bucket versioning is enabled, the "remove object" operation simply puts a delete marker on top. Keep in mind that the delete marker is not explicitly put by us. Whether there will be a delete marker depends on whether bucket versioning is enabled and on the method of removal.

There is nothing in Splunk about versioning. It's at the storage level. Splunk only does
1) "simple" object removal or
2) removal of all versions of an object, depending on the configuration.

Re: [SmartStore]Can you provide me with some clarification on remote.s3.supports_versioning configuration for SmartStor

pellegrini — Tue, 26 Mar 2024 19:14:06 GMT

I had this exact issue in one environment.

Versioning turned off in AWS S3 and turned on in Splunk. It works perfectly fine until a index bucket needs to freeze. Then Splunk is not able to remove any index bucket related files on S3, and splunkd will log errors and warnings. This event gives a hint of the issue:

03-26-2024 18:53:54.640 +0100 WARN S3Client [118080 FilesystemOpExecutorWorker-0] - Error removing object name=splunk01/index01/db/9c/b2/1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF/receipt.json(0,-1,0,) as versions of the object could not be listed

These are all events related to failing to freeze a index bucket on S3:

03-26-2024 18:53:54.640 +0100 INFO BucketMover [118080 FilesystemOpExecutorWorker-0] - RemoteStorageAsyncFreezer freeze completed succesfully for bid=index01_ccd~1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF 03-26-2024 18:53:54.640 +0100 WARN DatabaseDirectoryManager [118080 FilesystemOpExecutorWorker-0] - failed to request CacheManager to remove remote data for bucket, cache_id="bid|index01_ccd~1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF|", exception=Error removing bucket with cache_id="bid|index01_ccd~1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF|" 03-26-2024 18:53:54.640 +0100 ERROR CacheManager [118080 FilesystemOpExecutorWorker-0] - cache_id="bid|index01_ccd~1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF|", issue="Failed to remove receipt remoteId=splunk01/index01_ccd/db/9c/b2/1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF/receipt.json(0,-1,0,)" 03-26-2024 18:53:54.640 +0100 ERROR CacheManager [118080 FilesystemOpExecutorWorker-0] - Remove bucket cache_id="bid|index01_ccd~1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF|" with receiptId=splunk01/index01_ccd/db/9c/b2/1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF/receipt.json(0,-1,0,) failed 03-26-2024 18:53:54.640 +0100 WARN S3Client [118080 FilesystemOpExecutorWorker-0] - Error removing object name=splunk01/index01_ccd/db/9c/b2/1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF/receipt.json(0,-1,0,) as versions of the object could not be listed 03-26-2024 18:53:54.558 +0100 INFO CacheManager [118080 FilesystemOpExecutorWorker-0] - will remove cache_id="bid|index01_ccd~1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF|" from remote storage 03-26-2024 18:53:54.545 +0100 INFO CacheManager [118080 FilesystemOpExecutorWorker-0] - will remove cache_id="bid|index01_ccd~1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF|" removeRemote=1 03-26-2024 18:53:54.379 +0100 INFO BucketMover [118080 FilesystemOpExecutorWorker-0] - RemoteStorageAsyncFreezer trying to freeze bid=index01_ccd~1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF, freezeInitiatedByAnotherPeer=false 03-26-2024 18:53:54.379 +0100 INFO DatabaseDirectoryManager [118080 FilesystemOpExecutorWorker-0] - cache_id="bid|index01_ccd~1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF|" found to be on remote storage v