Getting Data In

Estimating size of blockSignature index

dwaddle
SplunkTrust
SplunkTrust

Is there any data available on estimating how big an indexer's blockSignature database may become, based the blockSignSize and daily event volumes?

Tags (2)

jrodman
Splunk Employee
Splunk Employee

blockSIgnature events are just listings of eventIDs. Therefore a block signature event is just a bunch of numbers like 45:35 with spaces in between. Therefore the size of the block signature database has no correspondence with the size of the original events, making this difficult to estimate in bytes.

However, each event requires a price of around 8 bytes in the text, precompressed, and each event requires a hash and indexing-related overhead. I would guesstimate that each block, in in the current rawdata format and tsidx format, is going to cost on the order of a kilobyte. Empirical data is obviously vastly superior to my guesses.

There's therefore the obvious tunable here which is the blockSignSize. I think anyone using this feature significantly probably wants to tune that value upwards. You don't want it too high for low incoming datarates, but at higher rates, plausibly values like 10000 might be reasonable.

0 Karma

Michael_Wilde
Splunk Employee
Splunk Employee

Good question. Mine is the same size of my main index. Not thrilled.

Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Best Practices: Splunk auto adjust pipeline queue

When you enable autoAdjustQueue in Splunk, maxSize should be understood as the queue size Splunk starts with ...

Introducing the 2026 - 2027 SplunkTrust cohort!

The goal of the SplunkTrust™ membership has historically been to acknowledge and recognize those who go above ...

Splunk Auto Ingestion Parallel Pipeline Scaling

Why this feature matters Many Splunk environments experience ingestion pressure long before the host is fully ...