We are using DBConnect with AML requirements. The retention period of splunk was 1 year. But it turned out to be necessary for seven years.
Therefore, I would like to ask three questions.
For deletion after the retention period, will it be judged by the import date and time? Or should it be judged by looking at the DB's timestamp?
Please tell us about the timing of deletion after the retention period. (Do you want to delete it immediately, or delete it regularly?)
I would like to refer to the deletion history in cloud.Please tell us the query to refer to.Or can you give me the deletion history?
Data is deleted by bucket shortly after the newest event in that bucket becomes older than the retention period. IOW, if a bucket contains events from 2020-06-11 until 2020-06-13 and the retention period is one year then the bucket will be deleted on 2021-06-14. Buckets are dated based on the _time field of the events within it. _time can be the DB time, the time of ingestion, or anything else.
Splunk checks every 60 seconds to see if there is a bucket that should be frozen.
Deleted buckets are recorded in the _internal index. Look for "BucketMover" events.
BTW, these retention behaviors are standard with Splunk and not specific to DB Connect. DB Connect has no effect on data retention.
Thanks richgalloway
I have one more question to ask.
Is the deletion process sure to delete all at once? Or maybe it can't be deleted, will it be deleted in the next deletion process? It seems that not all contents are deleted at once here.
When the BucketMover log was extracted from the Splunk Cloud logs, the following 4 types of logs were found in the _raw field. Could you tell me what kind of processing is performed by BucketMover and the log that is output?
No.1 Will freeze bkt=
No.2 RemoteStorageAsyncFreezer freeze skipped for bid=
No.3 RemoteStorageAsyncFreezer trying to freeze bid=
No.4 RemoteStorageAsyncFreezer freeze completed succesfully for bid=
The data that had exceeded the retention period was left undeleted. The number of each log for one month when the storage period is exceeded is as follows.
No.1 22,760
No.2 22,526
No. 3 234
No.4 234
In this case, is there a possibility that the data that has exceeded the retention period will remain without being deleted? Currently, the storage period has been extended, so no data is over the storage period. In that case, is the above log output? Even if it exceeds the limit, will a similar log be output? Or is there any change in the contents of the exceeded log?
Thanks for your answer. I asked for support.
Excuse me. Let me ask two more questions.
Is there a factor that data is deleted on the cloud other than the data retention period of index and the contracted data size? Please let me know if any.
Can you see the breakdown of the data deleted in BucketMover's log?