Hi @kn450 To address high storage utilization by moving older Splunk data, the recommended approach involves configuring data retirement policies. Manually moving buckets is generally discouraged d...
See more...
Hi @kn450 To address high storage utilization by moving older Splunk data, the recommended approach involves configuring data retirement policies. Manually moving buckets is generally discouraged due to complexity and risk. Implement Data Retention Policies: Configure your indexes.conf file to automatically manage data lifecycle (hot -> warm -> cold -> frozen). Set frozenTimePeriodInSecs to define when data should be considered frozen. Data in the frozen state is typically deleted by Splunk, but you can configure a script (coldToFrozenScript) to move it to external storage instead, or coldToFrozenDir for a frozen path on additional storage. However, searching this manually moved frozen data requires restoring / thawing before being searchable again by Splunk. Immediate Action (Use with Caution): If space is critical now and retention policies aren't configured: Identify the oldest cold buckets ($SPLUNK_DB/<index_name>/colddb/*). Backup these buckets first. Manually move the oldest cold buckets to external storage. This frees up space but makes the data unsearchable by Splunk unless restored. Alternatively, if data loss is acceptable for the oldest data, adjust frozenTimePeriodInSecs to a shorter duration and restart Splunk; it will begin freezing (and potentially deleting, depending on configuration) older data. This is irreversible if deletion is enabled. Accessing Migrated Data: Frozen data must be manually restored (thawed) back into a Splunk index's thawed directory (thaweddb) to be searched again. This is a manual process. - For more info please see https://docs.splunk.com/Documentation/Splunk/9.4.1/Indexer/Restorearchiveddata Splunk manages data through buckets representing time chunks. These buckets transition from hot (actively written), to warm (read-only), to cold (read-only, potentially moved). The final state is frozen, where Splunk expects the data to be archived or deleted based on indexes.conf settings. Manually moving buckets breaks this native searchability. For more info check out https://docs.splunk.com/Documentation/Splunk/9.4.1/Indexer/Automatearchiving Top Tips Backup: Always back up data before manually moving or deleting buckets. Configuration: Properly configuring indexes.conf (especially homePath, coldPath, thawedPath, maxTotalDataSizeMB, frozenTimePeriodInSecs) is crucial for managing storage automatically. Manual Migration Risk: Manually moving buckets is error-prone and complex to manage, especially for searching. It should be a last resort or temporary measure. Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing