topic Re: data trim in Security

data trim

Siddharthnegi — Tue, 08 Aug 2023 08:23:51 GMT

Thanks for your answer, however, we are facing an issue where there is enough space in our index but our disk space has reached around 80%. SO I just want to know if volume trimming happens on the disk level as well ? Below attached are our index configuration for paloalto index and the disk status.

[firewall_paloalto]
coldPath = volume:cold\firewall_paloalto\colddb
homePath = volume:hotwarm\firewall_paloalto\db
thawedPath = D:\splunk_data\firewall_paloalto\thaweddb
tstatsHomePath = volume:hotwarm\firewall_paloalto\datamodel_summary

frozenTimePeriodInSecs = 47304000

maxTotalDataSizeMB = 4294967295

Re: data trim

richgalloway — Tue, 08 Aug 2023 12:47:14 GMT

When buckets age out and are frozen (deleted) then disk space will be restored. The buckets need to be at least 1.5 years old before they will be deleted, however, given the frozenTimePeriodInSecs setting.

Buckets also will be deleted as needed to stay within the maxTotalDataSizeMB setting, but it may take a long time to fill 4PB (depending on your ingest rate).

You may want to confirm the settings are appropriate for the index.

Re: data trim

Siddharthnegi — Wed, 09 Aug 2023 05:14:14 GMT

Thanks for the answer , but the problem is we have enough storage for index but still its trimming data . And disk space is used around 80% , So i want to know whether volume trimming happens on the disk level as well.

Re: data trim

richgalloway — Wed, 09 Aug 2023 18:15:36 GMT

There are many settings that factor into when data is reaped, which makes it a bit complicated. It's further complicated if you use volumes or SmartStore.

Can you share the indexes.conf stanza for the index and the [default] indexes.conf stanza?

Re: data trim

Siddharthnegi — Thu, 10 Aug 2023 04:18:59 GMT

Are you talking about this?

frozenTimePeriodInSecs = 47304000

maxTotalDataSizeMB = 4294967295

Re: data trim

richgalloway — Thu, 10 Aug 2023 11:59:10 GMT

Is there also a [default] stanza in indexes.conf? What are the volume settings?

Re: data trim

Siddharthnegi — Thu, 10 Aug 2023 12:16:44 GMT

this is default stanza

[default]
enableDataIntegrityControl = true
frozenTimePeriodInSecs = 47304000
repFactor = auto
maxWarmDBCount = 80
maxTotalDataSizeMB = 4294967295

[volume:hotwarm]
path = /opt/index_data/splunk_data

[volume:cold]
path = /opt/index_data/splunk_data

[volume:tstats]
path = /opt/index_data/splunk_data_tstats

Re: data trim

richgalloway — Thu, 10 Aug 2023 16:01:33 GMT

The volume settings should include maxVolumeDataSizeMB so Splunk knows how large the volume is (or at least how much it can use). Each index can use individual maxTotalDataSizeMB settings to control how much of the volume they can consume.

Re: data trim

Siddharthnegi — Fri, 11 Aug 2023 12:09:21 GMT

Why the data is being trimmed if the index have enough space to store new data as well as old data?

Re: data trim

richgalloway — Fri, 11 Aug 2023 12:38:46 GMT

Perhaps, in the absence of maxVolumeDataSizeMB, Splunk is using a low value for the size of the volume and trimming data to "fit" that lower value.