Issue:
Buckets health indicator reports gigantic warm bucket for _metrics,
but the settled warm bucket is approximately 10 GiB. splunk enterprise 10.0.2 , indexer cluster with a search head was join to the indexer cluster. i have warning for _metric bucket more than 20GB.
Configuration:
datatype=metric
maxDataSize=auto_high_volume
metric.stubOutRawdataJournal=true
repFactor=0
Example: Indexer-02, bucket 337
HotBucketRoller:
bid=_metrics~337~5218DBAE-3E56-44C9-8729-872CF04D2A87
size=21993381888
caller=size_exceeded
_maxHotBucketSize=10737418240
bucketSize=10737451008
PeriodicHealthReporter:
gigantic_bucket_size
bucket_size=21993381888
yellow_size_threshold=20971520000
After rollover:
dbinspect:
bucket=337
state=warm
sizeOnDiskMB=10269.71
sizeGiB=10.029
Filesystem:
du -sh bucket/
11G
du -sh bucket/rawdata
12K
Observation:
The Health Reporter bucket_size is byte-for-byte identical to
HotBucketRoller's transient physical "size", but after
metric.stubOutRawdataJournal completes, the resulting warm bucket
is approximately 10 GiB.
Question:
Should the gigantic_bucket_size health check exclude the temporary
rawdata journal size for metric indexes where
metric.stubOutRawdataJournal=true?
Why does PeriodicHealthReporter continue reporting the same historical
23087370240 / 21993381888 values after the associated buckets have
settled to normal size?
