Deployment Architecture
Highlighted

Why are the bucket numbers in colddb out of sequence?

Explorer

I see in the db of one of my indexers:
drwx--x--- 3 root root 4096 Aug 25 22:29 db150377910015037008824044
drwx--x--- 3 root root 4096 Aug 26 05:05 db
150380280015037217504045
drwx--x--- 3 root root 4096 Aug 26 11:41 db150382680015037455004046
drwx--x--- 3 root root 4096 Aug 26 18:15 db
150385020015037692614047
......

In the colddb:
......
drwx--x--- 3 root root 4096 Aug 25 00:07 db150369870015036216714040
drwx--x--- 3 root root 4096 Aug 25 06:06 db
150372030015036412644041
drwx--x--- 3 root root 4096 Aug 25 11:35 db150373986015036627814042
drwx--x--- 3 root root 4096 Aug 25 16:41 db
150375840015036825004043
drwx--x--- 3 root root 4096 Sep 11 13:56 db149032369114862378363906
drwx--x--- 3 root root 4096 Sep 11 14:44 db
149032585114903231164115
drwx--x--- 3 root root 4096 Sep 11 15:52 db149033017114903258504117
drwx--x--- 3 root root 4096 Sep 21 17:49 db
149120135114903299444119
drwx--x--- 3 root root 4096 Nov 9 09:13 db14954110941491201516_4154

What can cause the rotation from warm bucket to cold bucket to have a big gap (Sep. 21 - Nov. 9) and the bucket rolling date and bucket number out of order. Before db14903236911486237836_3906, when listing by timestamp, all bucket numbers are in sequence. Should I be concerned?

Thanks.

0 Karma
Highlighted

Re: Why are the bucket numbers in colddb out of sequence?

SplunkTrust
SplunkTrust

Hi @yuelu,

Let me explain why there is sequence number out of order, by default splunk will create upto 3 hot buckets and these hot buckets can roll from hot to warm based on some of the parameter (maxHotSpanSecs and maxDataSize, whichever hit first). If we consider bucket db_1490323691_1486237836_3906 in this case earliest event is from 4th Feb 2017 19:50 GMT to latest event is 24th March 2017 02:48 GMT and if I assume that you have default maxHotSpanSecs which is 90 days then you didn't hit this parameter and I assume you didn't hit maxDataSize as well then in this case splunk will roll this bucket when you will have more than 3 hot buckets or when you will restart splunk.

To roll bucket from Warm to Cold is depend on maxWarmDBCount parameter. Here if we consider about bucket db_1495411094_1491201516_4154 which has earliest event from 3rd April 2017 06:38 to latest event 21st May 2017 23:58 GMT but why this bucket has id 4154, this occurs when splunk is not able to parse timestamp properly or your event contains old timestamp so splunk generate new bucket with new id in sequence but contains earliest time and latest time based on events which is present in that bucket so in this case you need to check whether forwarder itself sending events with old timestamp or splunk is not able to parse timestamp properly.

If you still require more info then please provide your indexes.conf configuration so someone from community can help you.

I hope this helps.

Thanks,
Harshil

0 Karma
Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.