Splunk Cloud Platform

Data retention period in DBConnect

nishida_tada_ca
Loves-to-Learn Lots

We are using DBConnect with AML requirements. The retention period of splunk was 1 year. But it turned out to be necessary for seven years.
Therefore, I would like to ask three questions.

For deletion after the retention period, will it be judged by the import date and time? Or should it be judged by looking at the DB's timestamp?

Please tell us about the timing of deletion after the retention period. (Do you want to delete it immediately, or delete it regularly?)

I would like to refer to the deletion history in cloud.Please tell us the query to refer to.Or can you give me the deletion history?

Labels (1)
Tags (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Data is deleted by bucket shortly after the newest event in that bucket becomes older than the retention period.  IOW, if a bucket contains events from 2020-06-11 until 2020-06-13 and the retention period is one year then the bucket will be deleted on 2021-06-14.  Buckets are dated based on the _time field of the events within it.  _time can be the DB time, the time of ingestion, or anything else.

Splunk checks every 60 seconds to see if there is a bucket that should be frozen.

Deleted buckets are recorded in the _internal index.  Look for "BucketMover" events.

BTW, these retention behaviors are standard with Splunk and not specific to DB Connect.  DB Connect has no effect on data retention.

---
If this reply helps you, Karma would be appreciated.
0 Karma

nishida_tada_ca
Loves-to-Learn Lots

Thanks richgalloway

I have one more question to ask.

Is the deletion process sure to delete all at once? Or maybe it can't be deleted, will it be deleted in the next deletion process? It seems that not all contents are deleted at once here.

0 Karma

richgalloway
SplunkTrust
SplunkTrust
Yes, the entire bucket is deleted at once. There should be no reason a bucket cannot be deleted, but if there is it will be retried a minute later.
---
If this reply helps you, Karma would be appreciated.
0 Karma

nishida_tada_ca
Loves-to-Learn Lots

When the BucketMover log was extracted from the Splunk Cloud logs, the following 4 types of logs were found in the _raw field. Could you tell me what kind of processing is performed by BucketMover and the log that is output?

No.1 Will freeze bkt=

No.2 RemoteStorageAsyncFreezer freeze skipped for bid=

No.3 RemoteStorageAsyncFreezer trying to freeze bid=

No.4 RemoteStorageAsyncFreezer freeze completed succesfully for bid=

The data that had exceeded the retention period was left undeleted. The number of each log for one month when the storage period is exceeded is as follows.

No.1 22,760

No.2 22,526

No. 3 234

No.4 234

In this case, is there a possibility that the data that has exceeded the retention period will remain without being deleted? Currently, the storage period has been extended, so no data is over the storage period. In that case, is the above log output? Even if it exceeds the limit, will a similar log be output? Or is there any change in the contents of the exceeded log?

0 Karma

richgalloway
SplunkTrust
SplunkTrust
Splunk will continue to try to delete buckets. I know of no log entry when storage limits are exceeded.
Consider opening a support request with Splunk Cloud on this.
---
If this reply helps you, Karma would be appreciated.
0 Karma

nishida_tada_ca
Loves-to-Learn Lots

Thanks for your answer. I asked for support.

0 Karma

nishida_tada_ca
Loves-to-Learn Lots

Excuse me. Let me ask two more questions.

Is there a factor that data is deleted on the cloud other than the data retention period of index and the contracted data size? Please let me know if any.

Can you see the breakdown of the data deleted in BucketMover's log?

0 Karma

richgalloway
SplunkTrust
SplunkTrust
Size and retention time are the only criteria for when data is deleted. Data can be deleted to make room for new data once an index reaches it's defined maximum size.
---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Splunk Education Goes to Washington | Splunk GovSummit 2024

If you’re in the Washington, D.C. area, this is your opportunity to take your career and Splunk skills to the ...