Archive
Highlighted

Hot to warm dbRoll

New Member

Yesterday db rolled from hot to warm, and today there is a new hot db but there isn't the warm db. In the file .bucketManifest there is the list of the hot and warm db but there is no warm db into the hd. Why splunk does'n create any warm db ? My operating system is windows 2003 server, my splunk release is 4.0

Thank you Francesco

Here the log i found:

04-27-2010 22:47:15.594 INFO  databasePartitionPolicy - Moving db with id of 2: D:\Program Files\Splunk\var\lib\splunk\defaultdb\db\hot_v1_2 to warm: size exceeded: maxDataSize=52428800 bytes, bucketSize=52430881 bytes
04-27-2010 22:47:15.609 WARN  databasePartitionPolicy - About to move db at D:\Program Files\Splunk\var\lib\splunk\defaultdb\db\hot_v1_2 to warm
04-27-2010 22:47:15.984 INFO  TPool - All 2 workers of IndexerTPool terminated
04-27-2010 22:47:15.984 INFO  TPool - initializing IndexerTPool with 2 workers
04-27-2010 22:47:16.375 INFO  timeinvertedIndex - started splunk-optimize-lex for dir D:\Program Files\Splunk\var\lib\splunk\defaultdb\db\db_1272401212_1271671603_2
04-27-2010 22:47:22.203 INFO  timeinvertedIndex - started recover-metadata for dir D:\Program Files\Splunk\var\lib\splunk\defaultdb\db\db_1272401212_1271671603_2
04-27-2010 22:47:22.453 INFO  BucketMover - will attempt to freeze: D:\Program Files\Splunk\var\lib\splunk\defaultdb\db\db_1272401212_1271671603_2 becuase maxTotalDataSize=52428800 bytes, currentSize=68640623 bytes
04-27-2010 22:47:22.469 INFO  BucketMover - terminating final optimizes for D:\Program Files\Splunk\var\lib\splunk\defaultdb\db\db_1272401212_1271671603_2 in preparation of move
04-27-2010 22:47:22.703 INFO  timeinvertedIndex - removed file D:\Program Files\Splunk\var\lib\splunk\defaultdb\db\db_1272401212_1271671603_2\1271983254-1271671603-102280.tsidx.lock
04-27-2010 22:47:22.922 INFO  timeinvertedIndex - removed file D:\Program Files\Splunk\var\lib\splunk\defaultdb\db\db_1272401212_1271671603_2\1271983262_1271671603_64440-1272375139_1271983254_102280-12916.merge
04-27-2010 22:47:22.922 INFO  timeinvertedIndex - removed file D:\Program Files\Splunk\var\lib\splunk\defaultdb\db\db_1272401212_1271671603_2\1272375139-1271983262-64440.tsidx.lock
04-27-2010 22:47:45.359 INFO  HotDBManager - no hot found for event ts=1272401264, closest match=null [expanded span=0]
04-27-2010 22:47:45.375 INFO  databasePartitionPolicy - creating new DB D:\Program Files\Splunk\var\lib\splunk\defaultdb\db\hot_v1_3
04-27-2010 22:47:45.375 INFO  timeinvertedIndex - Opening D:\Program Files\Splunk\var\lib\splunk\defaultdb\db\hot_v1_3
04-27-2010 22:47:45.375 INFO  timeinvertedIndex - No files to decompress on create
04-27-2010 22:47:45.375 INFO  timeinvertedIndex - create by dirname D:\Program Files\Splunk\var\lib\splunk\defaultdb\db\hot_v1_3
04-27-2010 22:47:45.437 INFO  databasePartitionPolicy - lazy loading database for: D:\Program Files\Splunk\var\lib\splunk\defaultdb\db\hot_v1_3, id=3, ts=1272401264 dirMgr::nextId=3]
04-27-2010 22:47:45.437 INFO  HotDBManager - creating new hot (id=3, time=1272401264)]
04-27-2010 22:47:45.469 WARN  databasePartitionPolicy - Adding D:\Program Files\Splunk\var\lib\splunk\defaultdb\db\hot_v1_3 because it was not in the manifest for metadata.
Tags (3)
0 Karma
Highlighted

Re: Hot to warm dbRoll

Super Champion

Looks like your issue is that your buckets are moving from hot --> warm --> frozen immediately. So if you don't have a coldToFrozenScript program setup (which is the default), then the default action is to purge the bucket.

Notice the log entry:

BucketMover - will attempt to freeze: D:\Program Files\Splunk\var\lib\splunk\defaultdb\db\db_1272401212_1271671603_2 becuase maxTotalDataSize=52428800 bytes, currentSize=68640623 bytes

If my math is right, the log is indicating that your total maximum index size for your default bucket is 50 Mb, which isn't much, and your bucket is 65Mb. So because 65>50, splunk froze your oldest bucket. (I'm guessing that "hot" buckets are excluded from this check, which is why they are removed right after being rolled to warm.)

You should be able to solve this by editing your indexes.conf file and setting maxTotalDataSizeMB to a larger value for your default (main) index. Splunk sets this to 500G out of the box.

The following two docs address this issue and contains links to other related content:

Highlighted

Re: Hot to warm dbRoll

Splunk Employee
Splunk Employee

Lowell is spot on. We need to send you a gift or something. Definitely review your maximum index size, as suggested.

0 Karma
Highlighted

Re: Hot to warm dbRoll

Splunk Employee
Splunk Employee

Seperately, responding to the comment about hot not being counted: Hot buckets aren't accounted for when considering the total index size, as far as I know. Support/sustaining view this as a bug, and it's assigned to be changed so they are included in the total size. However, we aren't currently planning to apply the accounting principles to hot buckets. In other words, hot buckets will never be curtailed early, until splunk sees fit to roll them to warm. That's the near-term plan.

0 Karma