Deployment Architecture

Configurations for a "baseline" + "updates" type index



I'm trying to figure out the most recommended way to set up an index that stores data ingested in the following manner:

1) Every ~30 days a baseline of events is sent, specifying the current "truth".

2) Between baselines, small updates are ingested, specifying diffs from the previous baseline.

A baseline would be around ~1 GB, and the small updates would be ~1 MB every few days.

Queries on this index will build a "current state" by querying the baseline + the updates since. This would require a baseline + updates to be kept in warm buckets. 

I was wondering what would be the best indexes.conf configuration for this case?

My initial thought was:

frozenTimePeriodInSecs=7776000  # 90 days to keep ~3 baselines
maxDataSize=2000 # max size of a baseline

The reason I set maxWarmDBCount to 30 was in case of an update every day, and automatic rolling from hot to warm bucket. If hot buckets can stay hot for multiple days, I could reduce this number.

Any inputs?




Labels (1)
0 Karma
Get Updates on the Splunk Community!

Observability | How to Think About Instrumentation Overhead (White Paper)

Novice observability practitioners are often overly obsessed with performance. They might approach ...

Cloud Platform | Get Resiliency in the Cloud Event (Register Now!)

IDC Report: Enterprises Gain Higher Efficiency and Resiliency With Migration to Cloud  Today many enterprises ...

The Great Resilience Quest: 10th Leaderboard Update

The tenth leaderboard update (11.23-12.05) for The Great Resilience Quest is out >> As our brave ...