Can I archive data in Splunk Cloud?

[adukes_splunk]

Since I don't have an on-premise storage option, how can I keep data beyond my 90 day retention allocation?

[adukes_splunk · Accepted solution]

The Splunk Product Best Practices team provided this response. Read more about How Crowdsourcing is Shaping the Future of Splunk Best Practices.

How indexes help you manage, store, and restore data

Indexes store the data sent to your Splunk Cloud deployment. You can create, update, delete, and view index properties, modify data retention settings for individual indexes, delete data from indexes, and optimize search performance by managing the number of indexes and the data sources stored in specific indexes. See manage Splunk Cloud indexes to learn best practices for indexes.

The Splunk Cloud service bases your storage space on the volume of uncompressed data you want to index daily and comes with enough storage to store up to 90 days of uncompressed data. For example, if your daily volume of uncompressed data is 100 GB, your Splunk Cloud has 9000 GB (9 TB) of storage. When the index reaches the specified days retention, Splunk Cloud deletes the oldest data from the index.

If you need to store data beyond your retention allocation and have a Managed Splunk Cloud, you can augment Splunk Cloud with Dynamic Data Self Storage (DDSS) or Dynamic Data Active Archive (DDAA). DDSS is available by default to Splunk Cloud customers. DDAA is a low-cost option to move your data to a Splunk-maintained searchable archive.

Splunk Cloud places the data you send in indexes you self-manage from the Indexes page in Splunk Web. Splunk Cloud retains data based on index settings that enable you to specify when to delete data. Review the Splunk Cloud data policies before you configure data retention settings for different data sources. Data is not searchable after Splunk Cloud deletes it from the index. It's a best practice to store data in separate indexes to meet your audit and compliance requirements.

Things to know

If you've configured DDAA or DDSS as data ages from searchable old, data automatically moves to the appropriate repository when the storage meets the retention setting for an index. The Splunk Cloud Monitoring (CMC) app is part of Splunk Cloud and is available to help you monitor Splunk Cloud deployment health. CMC displays details about your storage consumption and details such as data stored, number of days of retention for each index.

DDSS: Exports your oldest data to your AWS S3 account before deleting it from the index. Review the requirements for Dynamic Data Self Storage to see how to export of your aged, ingested data. Also see Configure self storage locations on Amazon S3 and Dynamic Data: Self-Storage – Compliance, Cloud and Data Lifecycle.

DDAA: Stores data until the retention setting that you specify expires. Your DDAA subscription entitles you to restore up to 10% of your archive subscription per restore. Restored data is searchable within 24 hours of the restore time and remains searchable for up to 30 days. If multiple restores overlap within a 30-day period, it accrues against the restore entitlement. See Dynamic Data: Data Retention Options in Splunk Cloud.

Things to do

• Learn to archive expired Splunk Cloud data. Review DDAA requirement and the procedure to configure archive settings for indexes.
  

• Compare managed Splunk Cloud to self-service Splunk cloud. Review the Splunk Cloud Service Details and see the FAQ for Splunk Cloud and Self-service Splunk Cloud FAQ.

• Create a Splunk Cloud index and manage data retention settings. Review how to manage Splunk Cloud indexes and create an cloud index and set up data retention.

• Learn more about the importance of data retention. Review What is the default retention time in Splunk Cloud and what is the cost of extending it? and Splunk Cloud service limits and constraints