Deployment Architecture

[SmartStore]Can you provide me with some clarification on remote.s3.supports_versioning configuration for SmartStore?

rbal_splunk
Splunk Employee
Splunk Employee

When a bucket exceeds the configured data retention time and the parameter remote.s3.supports_versioning=true, then my understanding is that SmartStore will put a delete marker on the corresponding bucket that gets frozen and this data/bucket is ignored by SmartStore for any subsequent searches.

I'm seeing that the bucket gets completely deleted with no delete marker. I wanted to make sure that there's no other configuration that needs to be done other than:

Enable versioning on the S3 bucket
Ensure that remote.s3.supports_versioning=true (default)

Tags (2)
0 Karma

rbal_splunk
Splunk Employee
Splunk Employee

Just to complete this discussion.

When remote.s3.supports_versioning = true , we iterate over all versions of an S3 object (file) and remove all versions. Otherwise, we do a simple remove on the object. This means that if set to true, all versions will be removed and the object contents are irretrievable.

If set to false, the behavior is as follows:
1) if bucket versioning is disabled, the object is simply gone forever;
2) if bucket versioning is enabled, the "remove object" operation simply puts a delete marker on top. Keep in mind that the delete marker is not explicitly put by us. Whether there will be a delete marker depends on whether bucket versioning is enabled and on the method of removal.

There is nothing in Splunk about versioning. It's at the storage level. Splunk only does
1) "simple" object removal or
2) removal of all versions of an object, depending on the configuration.

0 Karma

rbal_splunk
Splunk Employee
Splunk Employee

When versioning is enabled on the S3 bucket:

If s3.supports_versioning=true, then the bucket/data is deleted w/o delete markers.
If s3.supports_versioning=false, then data is deleted and the bucket/data contains delete markers

0 Karma

srajarat2
Path Finder

Isn't this opposite of what versioning is supposed to do?

As per Splunk's indexes.conf documentation from https://docs.splunk.com/Documentation/Splunk/8.0.0/Admin/Indexesconf it clearly says versioning is a means of keeping multiple variants of an object.

remote.s3.supports_versioning = <boolean>
* Specifies whether the remote storage supports versioning.
* Versioning is a means of keeping multiple variants of an object
  in the same bucket on the remote storage.
* Optional.
* Default: true

Hence I would expect a delete marker in place when an object is deleted. Can you clarify?

0 Karma

srajarat2
Path Finder

What happens when the versioning is not enabled on the S3 bucket? How does remote.s3.supports_versioning behave?

0 Karma

pellegrini
Path Finder

I had this exact issue in one environment.

Versioning turned off in AWS S3 and turned on in Splunk. It works perfectly fine until a index bucket needs to freeze. Then Splunk is not able to remove any index bucket related files on S3, and splunkd will log errors and warnings. This event gives a hint of the issue:

03-26-2024 18:53:54.640 +0100 WARN  S3Client [118080 FilesystemOpExecutorWorker-0] - Error removing object name=splunk01/index01/db/9c/b2/1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF/receipt.json(0,-1,0,) as versions of the object could not be listed

 

These are all events related to failing to freeze a index bucket on S3:

03-26-2024 18:53:54.640 +0100 INFO  BucketMover [118080 FilesystemOpExecutorWorker-0] - RemoteStorageAsyncFreezer freeze completed succesfully for bid=index01_ccd~1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF
03-26-2024 18:53:54.640 +0100 WARN  DatabaseDirectoryManager [118080 FilesystemOpExecutorWorker-0] - failed to request CacheManager to remove remote data for bucket, cache_id="bid|index01_ccd~1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF|", exception=Error removing bucket with cache_id="bid|index01_ccd~1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF|"
03-26-2024 18:53:54.640 +0100 ERROR CacheManager [118080 FilesystemOpExecutorWorker-0] - cache_id="bid|index01_ccd~1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF|", issue="Failed to remove receipt remoteId=splunk01/index01_ccd/db/9c/b2/1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF/receipt.json(0,-1,0,)"
03-26-2024 18:53:54.640 +0100 ERROR CacheManager [118080 FilesystemOpExecutorWorker-0] - Remove bucket cache_id="bid|index01_ccd~1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF|" with receiptId=splunk01/index01_ccd/db/9c/b2/1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF/receipt.json(0,-1,0,) failed
03-26-2024 18:53:54.640 +0100 WARN  S3Client [118080 FilesystemOpExecutorWorker-0] - Error removing object name=splunk01/index01_ccd/db/9c/b2/1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF/receipt.json(0,-1,0,) as versions of the object could not be listed
03-26-2024 18:53:54.558 +0100 INFO  CacheManager [118080 FilesystemOpExecutorWorker-0] - will remove cache_id="bid|index01_ccd~1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF|" from remote storage
03-26-2024 18:53:54.545 +0100 INFO  CacheManager [118080 FilesystemOpExecutorWorker-0] - will remove cache_id="bid|index01_ccd~1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF|" removeRemote=1
03-26-2024 18:53:54.379 +0100 INFO  BucketMover [118080 FilesystemOpExecutorWorker-0] - RemoteStorageAsyncFreezer trying to freeze bid=index01_ccd~1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF, freezeInitiatedByAnotherPeer=false
03-26-2024 18:53:54.379 +0100 INFO  DatabaseDirectoryManager [118080 FilesystemOpExecutorWorker-0] - cache_id="bid|index01_ccd~1058~09FD8FE0-DA2A-4698-BE4C-BC2CD5D92EFF|" found to be on remote storage
v

 

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...