Why do we see large variations of the data age for a certain index?

[danielbb]

We see the following for one index in the cluster master -

Why do we see these fluctuations for the data age among the eight indexers?

[solarboyz1]

Buckets get rolled based on the youngest event in the bucket.

Buckets can span minutes, hours, days, etc.. depending on how much data is received into the index and your index configuration.
Additionally, if you turn up a new datasource and its starts reading in old logs. Those old events will now be mixed with your new events.
Finally, if the clock settings on the systems are incorrect...this can lead to events getting old or future dates.

Long story short, when old events get mixed in a bucket with newer events, the old events have to wait till ALL events in the bucket age out.

[danielbb]

Makes sense - how can I check these buckets?

[solarboyz1]

|  dbinspect index=yourindex 

It will show the startEpoch and endEpoch for each bucket, among other things.
https://docs.splunk.com/Documentation/Splunk/7.3.1/SearchReference/Dbinspect

If you want to drill into the events in a specific bucket:

index=yourindex | where _bkt="summary~22~4F582768-7B38-4768-95EA-EC3D491A8A23"

Where _bkt matches the bucketId from dbinspect results.

https://docs.splunk.com/Documentation/Splunk/7.3.1/Knowledge/Usedefaultfields