topic Re: Data trim in Knowledge Management

Data trim

Siddharthnegi — Fri, 04 Aug 2023 04:56:25 GMT

I have some questions regarding data trim

Re: Data trim

VatsalJagani — Fri, 04 Aug 2023 06:44:41 GMT

@Siddharthnegi - Please ask your question along with proper details so Splunk Community can help you.

Community Moderator,

Vatsal Jagani

Re: Data trim

Siddharthnegi — Fri, 04 Aug 2023 06:46:38 GMT

I want to ask from which version of splunk data trim feature has been added.

Re: Data trim

VatsalJagani — Fri, 04 Aug 2023 06:49:13 GMT

@Siddharthnegi - What do you mean by the splunk data trim? Can you describe it? Where do you it, in query or somewhere else?

Re: Data trim

gcusello — Fri, 04 Aug 2023 06:58:51 GMT

Hi @Siddharthnegi,

please see my answer to your previous question: https://community.splunk.com/t5/Splunk-Search/Searching-Cisco-ASA-sourcetype-in-Splunk/m-p/653115#M225689

Ciao.

Giuseppe

Re: Data trim

Siddharthnegi — Fri, 04 Aug 2023 06:59:01 GMT

yes,

so you allocate some storage to index and a retention period. now if that storage fills up before the retention period is over splunk intelligence starts to trim old data so that new data can be stored.

lets say you have allocated 500gb to particular index for 6 months retention time. And everyday 70gb data is being produced and stored in that index, now in less than 10 days storage will be filled so splunk intelligence then trim cold data from cold bucket.

Re: Data trim

gcusello — Fri, 04 Aug 2023 07:07:22 GMT

Hi @Siddharthnegi,

this means that you have to redesign your storage dimension because ingesting 70 GB/day and wanting to retain them for 180 days, you need:

Index_size = daily_ingestion * reduction_factor * retention_days

in other words:

Index_size = 70 * 0.5 * 180 = 6300 GB

you can also divide your storage:

Hot and Warm Buckets in an high performance storage (at least 800 IOPS)
Cold buckets in a less performant and expensive storage

Ciao.

Giuseppe