Splunk Dev

Create bloom filter after the fact

supersleepwalke
Communicator

I've been backfilling a year worth of logs, and just now realized that I didn't reconfigure maxBloomBackfillBucketAge, and none of these old logs have bloom filters, which is desperately necessary given the size of these logs. Is there any way I can create the bloom filters without having to blow these logs away and start from scratch?

0 Karma
1 Solution

supersleepwalke
Communicator

From index.conf docs:
http://docs.splunk.com/Documentation/Splunk/latest/Admin/indexesconf


maxBloomBackfillBucketAge = [smhd]
* If a (warm or cold) bucket is older than this, we shall not [re]create its blomfilter when we come across it
* Defaults to 30d.
* When set to 0, bloomfilters are never rebuilt

If you set this to a large number (e.g. 700d), and restart Splunk, it will automatically start recreating the bloom filters as part of the fsck process:

5-08-2012 09:54:33.066 -0500 INFO  ProcessTracker - (child_2__Fsck)  Fsck
- Rebuild --bloom-only bucket /opt/splunk/var/lib/splunk/proxy/db/db_1327467837_1327451635_11 took 2635.6 milliseconds 05-08-2012 09:55:05.173 -0500 INFO  ProcessTracker - (child_3__Fsck)  Fsck
- Rebuild --bloom-only bucket /opt/splunk/var/lib/splunk/proxy/db/db_1327451634_1327435722_10 took 3.535 seconds 05-08-2012 09:55:19.568 -0500 INFO  ProcessTracker - (child_4__Fsck)  Fsck
- Rebuild --bloom-only bucket /opt/splunk/var/lib/splunk/proxy/db/db_1327435721_1327426983_9 took 3.306 seconds

View solution in original post

Drainy
Champion

Probably an easier way to do this without editing configs would be to run the fsck rebuild process manually as per;

http://docs.splunk.com/Documentation/Splunk/4.3.2/admin/HowSplunkstoresindexes#Troubleshoot_your_buc...

supersleepwalke
Communicator

From index.conf docs:
http://docs.splunk.com/Documentation/Splunk/latest/Admin/indexesconf


maxBloomBackfillBucketAge = [smhd]
* If a (warm or cold) bucket is older than this, we shall not [re]create its blomfilter when we come across it
* Defaults to 30d.
* When set to 0, bloomfilters are never rebuilt

If you set this to a large number (e.g. 700d), and restart Splunk, it will automatically start recreating the bloom filters as part of the fsck process:

5-08-2012 09:54:33.066 -0500 INFO  ProcessTracker - (child_2__Fsck)  Fsck
- Rebuild --bloom-only bucket /opt/splunk/var/lib/splunk/proxy/db/db_1327467837_1327451635_11 took 2635.6 milliseconds 05-08-2012 09:55:05.173 -0500 INFO  ProcessTracker - (child_3__Fsck)  Fsck
- Rebuild --bloom-only bucket /opt/splunk/var/lib/splunk/proxy/db/db_1327451634_1327435722_10 took 3.535 seconds 05-08-2012 09:55:19.568 -0500 INFO  ProcessTracker - (child_4__Fsck)  Fsck
- Rebuild --bloom-only bucket /opt/splunk/var/lib/splunk/proxy/db/db_1327435721_1327426983_9 took 3.306 seconds
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...