https://community.splunk.com/t5/Deployment-Architecture/Hot-buckets-appear-to-be-growing-faster-than-they-can-be/m-p/75609#M2606 <P>I don't know that the message refers to hot buckets. I would take a look at the actual files on disk. Go to<BR /> <CODE>$SPLUNK_DB</CODE> (in Linux, the default is <CODE>/opt/splunk/var/lib/splunk</CODE>). You should see one directory for each index, along with some other files; note that the directory for the <CODE>main</CODE> index is called <CODE>defaultdb</CODE>. Take a look at the directory tree. Also check out the size of the <CODE>db</CODE> subdirectory and the <CODE>colddb</CODE> subdirectory. <CODE>db</CODE> holds the hot and warm buckets; <CODE>colddb</CODE> holds the cold buckets.</P> <P>Does the overall size of the directories match what you have in <CODE>indexes.conf</CODE>? If you look at these directories over time, can you see the buckets moving from <CODE>db</CODE> to <CODE>colddb</CODE> and then aging out? I would also check the maximum size setting for your indexes (you can do this in the Splunk Manager GUI if you prefer). Is the overall size of your index large enough? If the maximum size is too small, then Splunk may be aging out data sooner than you like.</P> <P>If you have a number of indexes and a high volume of data, buckets may be aging out of the indexes fairly quickly. That could be normal.</P> <P>Here is the documentation on <A href="http://docs.splunk.com/Documentation/Splunk/5.0.4/Indexer/HowSplunkstoresindexes">How Splunk Stores Indexes</A></P> Fri, 27 Sep 2013 16:37:30 GMT lguinn2 2013-09-27T16:37:30Z https://community.splunk.com/t5/Deployment-Architecture/Hot-buckets-appear-to-be-growing-faster-than-they-can-be/m-p/75608#M2605 <P>I have lots of lines like the ones below from <CODE>splunkd.log</CODE>:</P> <PRE><CODE>09-27-2013 15:04:28.681 +0000 INFO IndexProcessor - Starting to move buckets with the oldest latest time until we achieve compliance (current size=337945007355, max=146800640000) 09-27-2013 15:04:29.726 +0000 INFO IndexProcessor - Starting to move buckets with the oldest latest time until we achieve compliance (current size=337945445225, max=146800640000) 09-27-2013 15:04:30.761 +0000 INFO IndexProcessor - Starting to move buckets with the oldest latest time until we achieve compliance (current size=337948447673, max=146800640000) </CODE></PRE> <P>Doing some basic math shows that the <CODE>current size</CODE> only ever increases. Does this mean that I'm accumulating buckets faster than they can be aged off? What should I do?</P> Fri, 27 Sep 2013 15:16:35 GMT https://community.splunk.com/t5/Deployment-Architecture/Hot-buckets-appear-to-be-growing-faster-than-they-can-be/m-p/75608#M2605 markcaudill 2013-09-27T15:16:35Z https://community.splunk.com/t5/Deployment-Architecture/Hot-buckets-appear-to-be-growing-faster-than-they-can-be/m-p/75609#M2606 <P>I don't know that the message refers to hot buckets. I would take a look at the actual files on disk. Go to<BR /> <CODE>$SPLUNK_DB</CODE> (in Linux, the default is <CODE>/opt/splunk/var/lib/splunk</CODE>). You should see one directory for each index, along with some other files; note that the directory for the <CODE>main</CODE> index is called <CODE>defaultdb</CODE>. Take a look at the directory tree. Also check out the size of the <CODE>db</CODE> subdirectory and the <CODE>colddb</CODE> subdirectory. <CODE>db</CODE> holds the hot and warm buckets; <CODE>colddb</CODE> holds the cold buckets.</P> <P>Does the overall size of the directories match what you have in <CODE>indexes.conf</CODE>? If you look at these directories over time, can you see the buckets moving from <CODE>db</CODE> to <CODE>colddb</CODE> and then aging out? I would also check the maximum size setting for your indexes (you can do this in the Splunk Manager GUI if you prefer). Is the overall size of your index large enough? If the maximum size is too small, then Splunk may be aging out data sooner than you like.</P> <P>If you have a number of indexes and a high volume of data, buckets may be aging out of the indexes fairly quickly. That could be normal.</P> <P>Here is the documentation on <A href="http://docs.splunk.com/Documentation/Splunk/5.0.4/Indexer/HowSplunkstoresindexes">How Splunk Stores Indexes</A></P> Fri, 27 Sep 2013 16:37:30 GMT https://community.splunk.com/t5/Deployment-Architecture/Hot-buckets-appear-to-be-growing-faster-than-they-can-be/m-p/75609#M2606 lguinn2 2013-09-27T16:37:30Z