I see in the db of one of my indexers:
drwx--x--- 3 root root 4096 Aug 25 22:29 db_1503779100_1503700882_4044
drwx--x--- 3 root root 4096 Aug 26 05:05 db_1503802800_1503721750_4045
drwx--x--- 3 root root 4096 Aug 26 11:41 db_1503826800_1503745500_4046
drwx--x--- 3 root root 4096 Aug 26 18:15 db_1503850200_1503769261_4047
What can cause the rotation from warm bucket to cold bucket to have a big gap (Sep. 21 - Nov. 9) and the bucket rolling date and bucket number out of order. Before db_1490323691_1486237836_3906, when listing by timestamp, all bucket numbers are in sequence. Should I be concerned?
Let me explain why there is sequence number out of order, by default splunk will create upto 3 hot buckets and these hot buckets can roll from hot to warm based on some of the parameter (maxHotSpanSecs and maxDataSize, whichever hit first). If we consider bucket db_1490323691_1486237836_3906 in this case earliest event is from 4th Feb 2017 19:50 GMT to latest event is 24th March 2017 02:48 GMT and if I assume that you have default maxHotSpanSecs which is 90 days then you didn't hit this parameter and I assume you didn't hit maxDataSize as well then in this case splunk will roll this bucket when you will have more than 3 hot buckets or when you will restart splunk.
To roll bucket from Warm to Cold is depend on maxWarmDBCount parameter. Here if we consider about bucket db_1495411094_1491201516_4154 which has earliest event from 3rd April 2017 06:38 to latest event 21st May 2017 23:58 GMT but why this bucket has id 4154, this occurs when splunk is not able to parse timestamp properly or your event contains old timestamp so splunk generate new bucket with new id in sequence but contains earliest time and latest time based on events which is present in that bucket so in this case you need to check whether forwarder itself sending events with old timestamp or splunk is not able to parse timestamp properly.
If you still require more info then please provide your indexes.conf configuration so someone from community can help you.