What exactly does maxHotSpanSecs setting mean and why/when is it not honored ?

[hmahendrakumar]

I set maxHotSpanSecs=7200 and tried indexing a old log file containing a days worth of logs(~250mb 1 million events).
As per my understanding, it should create a bucket for every 2 hours worth of logs it indexes. But for some reason, it indexed all of it in a single bucket.

Am i missing something here ? Or have i misunderstood the definition ?

[sowings]

There are a couple of possible reasons I can think of. The bucket rotation doesn't occur immediately. It typically happens on an interval, as set in indexes.conf, rotatePeriodInSecs. It defaults to 60. So if your 250M file was indexed in less than a minute, then it might not have rotated. Furthermore, setting that parameter to 86400 or less introduces the "snapping" feature, meaning that it will want to round to the nearest day / hour or in your case, two hours. So when the time came to check for rotation, if the data that it had already indexed still fit within that two hour window, or if it had already all been indexed, or if the times were from so long ago that the bucket was a quarantine bucket....

To be clear, I have used maxHotSpanSecs set to 3600 during normal operation (i.e., just letting logs flow in, no historical indexing), and observed the exact behavior described in the docs: buckets span hours, from :00 minutes on the clock.

I suspect that soon after indexing of that old log file was complete, the bucket was rolled from hot to warm. I don't think that setting it to 7200 is a guarantee that hot buckets will only ever contain two hours' worth of data, particularly when doing batch or oneshot inputs.

[hmahendrakumar]

I have set rotatePeriodInSecs=10 , serviceMetaPeriod =10 and still see the same behavior.
Moreover, It still has not rolled over to warm. I see a single hot bucket.