Yesterday db rolled from hot to warm, and today there is a new hot db but there isn't the warm db. In the file .bucketManifest there is the list of the hot and warm db but there is no warm db into the hd. Why splunk does'n create any warm db ? My operating system is windows 2003 server, my splunk release is 4.0
Thank you Francesco
Here the log i found:
04-27-2010 22:47:15.594 INFO databasePartitionPolicy - Moving db with id of 2: D:\Program Files\Splunk\var\lib\splunk\defaultdb\db\hot_v1_2 to warm: size exceeded: maxDataSize=52428800 bytes, bucketSize=52430881 bytes 04-27-2010 22:47:15.609 WARN databasePartitionPolicy - About to move db at D:\Program Files\Splunk\var\lib\splunk\defaultdb\db\hot_v1_2 to warm 04-27-2010 22:47:15.984 INFO TPool - All 2 workers of IndexerTPool terminated 04-27-2010 22:47:15.984 INFO TPool - initializing IndexerTPool with 2 workers 04-27-2010 22:47:16.375 INFO timeinvertedIndex - started splunk-optimize-lex for dir D:\Program Files\Splunk\var\lib\splunk\defaultdb\db\db_1272401212_1271671603_2 04-27-2010 22:47:22.203 INFO timeinvertedIndex - started recover-metadata for dir D:\Program Files\Splunk\var\lib\splunk\defaultdb\db\db_1272401212_1271671603_2 04-27-2010 22:47:22.453 INFO BucketMover - will attempt to freeze: D:\Program Files\Splunk\var\lib\splunk\defaultdb\db\db_1272401212_1271671603_2 becuase maxTotalDataSize=52428800 bytes, currentSize=68640623 bytes 04-27-2010 22:47:22.469 INFO BucketMover - terminating final optimizes for D:\Program Files\Splunk\var\lib\splunk\defaultdb\db\db_1272401212_1271671603_2 in preparation of move 04-27-2010 22:47:22.703 INFO timeinvertedIndex - removed file D:\Program Files\Splunk\var\lib\splunk\defaultdb\db\db_1272401212_1271671603_2\1271983254-1271671603-102280.tsidx.lock 04-27-2010 22:47:22.922 INFO timeinvertedIndex - removed file D:\Program Files\Splunk\var\lib\splunk\defaultdb\db\db_1272401212_1271671603_2\1271983262_1271671603_64440-1272375139_1271983254_102280-12916.merge 04-27-2010 22:47:22.922 INFO timeinvertedIndex - removed file D:\Program Files\Splunk\var\lib\splunk\defaultdb\db\db_1272401212_1271671603_2\1272375139-1271983262-64440.tsidx.lock 04-27-2010 22:47:45.359 INFO HotDBManager - no hot found for event ts=1272401264, closest match=null [expanded span=0] 04-27-2010 22:47:45.375 INFO databasePartitionPolicy - creating new DB D:\Program Files\Splunk\var\lib\splunk\defaultdb\db\hot_v1_3 04-27-2010 22:47:45.375 INFO timeinvertedIndex - Opening D:\Program Files\Splunk\var\lib\splunk\defaultdb\db\hot_v1_3 04-27-2010 22:47:45.375 INFO timeinvertedIndex - No files to decompress on create 04-27-2010 22:47:45.375 INFO timeinvertedIndex - create by dirname D:\Program Files\Splunk\var\lib\splunk\defaultdb\db\hot_v1_3 04-27-2010 22:47:45.437 INFO databasePartitionPolicy - lazy loading database for: D:\Program Files\Splunk\var\lib\splunk\defaultdb\db\hot_v1_3, id=3, ts=1272401264 dirMgr::nextId=3] 04-27-2010 22:47:45.437 INFO HotDBManager - creating new hot (id=3, time=1272401264)] 04-27-2010 22:47:45.469 WARN databasePartitionPolicy - Adding D:\Program Files\Splunk\var\lib\splunk\defaultdb\db\hot_v1_3 because it was not in the manifest for metadata.
Looks like your issue is that your buckets are moving from
frozen immediately. So if you don't have a
coldToFrozenScript program setup (which is the default), then the default action is to purge the bucket.
Notice the log entry:
BucketMover - will attempt to freeze: D:\Program Files\Splunk\var\lib\splunk\defaultdb\db\db_1272401212_1271671603_2 becuase maxTotalDataSize=52428800 bytes, currentSize=68640623 bytes
If my math is right, the log is indicating that your total maximum index size for your default bucket is 50 Mb, which isn't much, and your bucket is 65Mb. So because 65>50, splunk froze your oldest bucket. (I'm guessing that "hot" buckets are excluded from this check, which is why they are removed right after being rolled to warm.)
You should be able to solve this by editing your
indexes.conf file and setting
maxTotalDataSizeMB to a larger value for your default (main) index. Splunk sets this to 500G out of the box.
The following two docs address this issue and contains links to other related content:
Seperately, responding to the comment about hot not being counted: Hot buckets aren't accounted for when considering the total index size, as far as I know. Support/sustaining view this as a bug, and it's assigned to be changed so they are included in the total size. However, we aren't currently planning to apply the accounting principles to hot buckets. In other words, hot buckets will never be curtailed early, until splunk sees fit to roll them to warm. That's the near-term plan.