When a bucket exceeds the configured data retention time and the parameter remote.s3.supports_versioning=true, then my understanding is that SmartStore will put a delete marker on the corresponding bucket that gets frozen and this data/bucket is ignored by SmartStore for any subsequent searches.
I'm seeing that the bucket gets completely deleted with no delete marker. I wanted to make sure that there's no other configuration that needs to be done other than:
Enable versioning on the S3 bucket
Ensure that remote.s3.supports_versioning=true (default)
When versioning is enabled on the S3 bucket:
If s3.supportsversioning=true, then the bucket/data is deleted w/o delete markers.
If s3.supportsversioning=false, then data is deleted and the bucket/data contains delete markers
Isn't this opposite of what versioning is supposed to do?
As per Splunk's indexes.conf documentation from https://docs.splunk.com/Documentation/Splunk/8.0.0/Admin/Indexesconf it clearly says versioning is a means of keeping multiple variants of an object.
remote.s3.supports_versioning = <boolean> * Specifies whether the remote storage supports versioning. * Versioning is a means of keeping multiple variants of an object in the same bucket on the remote storage. * Optional. * Default: true
Hence I would expect a delete marker in place when an object is deleted. Can you clarify?
What happens when the versioning is not enabled on the S3 bucket? How does remote.s3.supports_versioning behave?
Just to complete this discussion.
When remote.s3.supports_versioning = true , we iterate over all versions of an S3 object (file) and remove all versions. Otherwise, we do a simple remove on the object. This means that if set to true, all versions will be removed and the object contents are irretrievable.
If set to false, the behavior is as follows:
1) if bucket versioning is disabled, the object is simply gone forever;
2) if bucket versioning is enabled, the "remove object" operation simply puts a delete marker on top. Keep in mind that the delete marker is not explicitly put by us. Whether there will be a delete marker depends on whether bucket versioning is enabled and on the method of removal.
There is nothing in Splunk about versioning. It's at the storage level. Splunk only does
1) "simple" object removal or
2) removal of all versions of an object, depending on the configuration.