Hi,
I'm trying to figure out the most recommended way to set up an index that stores data ingested in the following manner:
1) Every ~30 days a baseline of events is sent, specifying the current "truth".
2) Between baselines, small updates are ingested, specifying diffs from the previous baseline.
A baseline would be around ~1 GB, and the small updates would be ~1 MB every few days.
Queries on this index will build a "current state" by querying the baseline + the updates since. This would require a baseline + updates to be kept in warm buckets.
I was wondering what would be the best indexes.conf configuration for this case?
My initial thought was:
frozenTimePeriodInSecs=7776000 # 90 days to keep ~3 baselines
maxDataSize=2000 # max size of a baseline
maxWarmDBCount=30
The reason I set maxWarmDBCount to 30 was in case of an update every day, and automatic rolling from hot to warm bucket. If hot buckets can stay hot for multiple days, I could reduce this number.
Any inputs?
Thanks!
