Deployment Architecture

Hot buckets appear to be growing faster than they can be migrated?

markcaudill
Engager

I have lots of lines like the ones below from splunkd.log:

09-27-2013 15:04:28.681 +0000 INFO  IndexProcessor - Starting to move buckets with the oldest latest time until we achieve compliance (current size=337945007355, max=146800640000)
09-27-2013 15:04:29.726 +0000 INFO  IndexProcessor - Starting to move buckets with the oldest latest time until we achieve compliance (current size=337945445225, max=146800640000)
09-27-2013 15:04:30.761 +0000 INFO  IndexProcessor - Starting to move buckets with the oldest latest time until we achieve compliance (current size=337948447673, max=146800640000)

Doing some basic math shows that the current size only ever increases. Does this mean that I'm accumulating buckets faster than they can be aged off? What should I do?

Tags (4)
0 Karma

lguinn2
Legend

I don't know that the message refers to hot buckets. I would take a look at the actual files on disk. Go to
$SPLUNK_DB (in Linux, the default is /opt/splunk/var/lib/splunk). You should see one directory for each index, along with some other files; note that the directory for the main index is called defaultdb. Take a look at the directory tree. Also check out the size of the db subdirectory and the colddb subdirectory. db holds the hot and warm buckets; colddb holds the cold buckets.

Does the overall size of the directories match what you have in indexes.conf? If you look at these directories over time, can you see the buckets moving from db to colddb and then aging out? I would also check the maximum size setting for your indexes (you can do this in the Splunk Manager GUI if you prefer). Is the overall size of your index large enough? If the maximum size is too small, then Splunk may be aging out data sooner than you like.

If you have a number of indexes and a high volume of data, buckets may be aging out of the indexes fairly quickly. That could be normal.

Here is the documentation on How Splunk Stores Indexes

0 Karma
Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...