Splunk Dev

how to set 3 month searchable data retention on a high volume per day index, currently only seeing 2 weeks worth?

u788332
New Member

maxDataSize if set to auto the default is 750 MB and 10GB on 64 bit and 1GB on 32 bit hosts, if it is set to auto_high_volume, is that correct?

frozenTimePeriodInSecs = 8640000
maxWarmDBCount = 50
maxHotBuckets = 55
maxDataSize = auto
So going by the configuration for this index , the calculations would come to around (50+55)*750MB = 76GB, also even if we go by 10 GB it would come to (50+55)*10GB = 1050GB for searchable data.
monthly usage for this index, equates to 2906.571GB, if we go by (default 10GB bucket size) then searchable data would be just 1050 GB that does fit the observed two week retention.
how much would we need to increase the maxWarmDBCount to? To allow for Data to be searchable still for 100 days?

thanks in advance

Tags (1)
0 Karma

jkat54
SplunkTrust
SplunkTrust

Your fronzenTimePeriodInSecs is 100 days... 86400 (secs/day) * 100 = 8640000

Try this for 30 days retention:

Add 1 sec to avoid ohSnap

[indexName]
maxHotSpanSecs = 86401 #not 86400 to avoid ohSnap
maxHotIdleSecs = 86401
frozenTimePeriodInSecs = 2592000
...

This will make sure the hot buckets close at the end of a day instead of waiting for the hot buckets to fill before they roll.

The net effect is buckets from 31 days ago are rolled to frozen.

Get Updates on the Splunk Community!

See just what you’ve been missing | Observability tracks at Splunk University

Looking to sharpen your observability skills so you can better understand how to collect and analyze data from ...

Weezer at .conf25? Say it ain’t so!

Hello Splunkers, The countdown to .conf25 is on-and we've just turned up the volume! We're thrilled to ...

How SC4S Makes Suricata Logs Ingestion Simple

Network security monitoring has become increasingly critical for organizations of all sizes. Splunk has ...