[SmartStore]Can you provide me with some clarification on remote.s3.supports_versioning configuration for SmartStore?

[rbal_splunk]

When a bucket exceeds the configured data retention time and the parameter remote.s3.supports_versioning=true, then my understanding is that SmartStore will put a delete marker on the corresponding bucket that gets frozen and this data/bucket is ignored by SmartStore for any subsequent searches.

I'm seeing that the bucket gets completely deleted with no delete marker. I wanted to make sure that there's no other configuration that needs to be done other than:

Enable versioning on the S3 bucket
Ensure that remote.s3.supports_versioning=true (default)

[rbal_splunk]

Just to complete this discussion.

When remote.s3.supports_versioning = true , we iterate over all versions of an S3 object (file) and remove all versions. Otherwise, we do a simple remove on the object. This means that if set to true, all versions will be removed and the object contents are irretrievable.

If set to false, the behavior is as follows:
1) if bucket versioning is disabled, the object is simply gone forever;
2) if bucket versioning is enabled, the "remove object" operation simply puts a delete marker on top. Keep in mind that the delete marker is not explicitly put by us. Whether there will be a delete marker depends on whether bucket versioning is enabled and on the method of removal.

There is nothing in Splunk about versioning. It's at the storage level. Splunk only does
1) "simple" object removal or
2) removal of all versions of an object, depending on the configuration.

[rbal_splunk]

When versioning is enabled on the S3 bucket:

If s3.supports_versioning=true, then the bucket/data is deleted w/o delete markers.
If s3.supports_versioning=false, then data is deleted and the bucket/data contains delete markers

[srajarat2]

Isn't this opposite of what versioning is supposed to do?

As per Splunk's indexes.conf documentation from https://docs.splunk.com/Documentation/Splunk/8.0.0/Admin/Indexesconf it clearly says versioning is a means of keeping multiple variants of an object.

remote.s3.supports_versioning = <boolean>
* Specifies whether the remote storage supports versioning.
* Versioning is a means of keeping multiple variants of an object
  in the same bucket on the remote storage.
* Optional.
* Default: true

Hence I would expect a delete marker in place when an object is deleted. Can you clarify?

[srajarat2]

What happens when the versioning is not enabled on the S3 bucket? How does remote.s3.supports_versioning behave?

[pellegrini]

I had this exact issue in one environment.

Versioning turned off in AWS S3 and turned on in Splunk. It works perfectly fine until a index bucket needs to freeze. Then Splunk is not able to remove any index bucket related files on S3, and splunkd will log errors and warnings. This event gives a hint of the issue:

03-26-2024 18:53:54.640 +0100 WARN  S3Client [118080 FilesystemOpExecutorWorker-0] - Error removing object name=splunk01/index01/db/9c/b2/1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF/receipt.json(0,-1,0,) as versions of the object could not be listed

These are all events related to failing to freeze a index bucket on S3:

03-26-2024 18:53:54.640 +0100 INFO  BucketMover [118080 FilesystemOpExecutorWorker-0] - RemoteStorageAsyncFreezer freeze completed succesfully for bid=index01_ccd~1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF
03-26-2024 18:53:54.640 +0100 WARN  DatabaseDirectoryManager [118080 FilesystemOpExecutorWorker-0] - failed to request CacheManager to remove remote data for bucket, cache_id="bid|index01_ccd~1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF|", exception=Error removing bucket with cache_id="bid|index01_ccd~1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF|"
03-26-2024 18:53:54.640 +0100 ERROR CacheManager [118080 FilesystemOpExecutorWorker-0] - cache_id="bid|index01_ccd~1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF|", issue="Failed to remove receipt remoteId=splunk01/index01_ccd/db/9c/b2/1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF/receipt.json(0,-1,0,)"
03-26-2024 18:53:54.640 +0100 ERROR CacheManager [118080 FilesystemOpExecutorWorker-0] - Remove bucket cache_id="bid|index01_ccd~1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF|" with receiptId=splunk01/index01_ccd/db/9c/b2/1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF/receipt.json(0,-1,0,) failed
03-26-2024 18:53:54.640 +0100 WARN  S3Client [118080 FilesystemOpExecutorWorker-0] - Error removing object name=splunk01/index01_ccd/db/9c/b2/1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF/receipt.json(0,-1,0,) as versions of the object could not be listed
03-26-2024 18:53:54.558 +0100 INFO  CacheManager [118080 FilesystemOpExecutorWorker-0] - will remove cache_id="bid|index01_ccd~1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF|" from remote storage
03-26-2024 18:53:54.545 +0100 INFO  CacheManager [118080 FilesystemOpExecutorWorker-0] - will remove cache_id="bid|index01_ccd~1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF|" removeRemote=1
03-26-2024 18:53:54.379 +0100 INFO  BucketMover [118080 FilesystemOpExecutorWorker-0] - RemoteStorageAsyncFreezer trying to freeze bid=index01_ccd~1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF, freezeInitiatedByAnotherPeer=false
03-26-2024 18:53:54.379 +0100 INFO  DatabaseDirectoryManager [118080 FilesystemOpExecutorWorker-0] - cache_id="bid|index01_ccd~1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF|" found to be on remote storage
v