Getting Data In

Estimating size of blockSignature index

dwaddle
SplunkTrust
SplunkTrust

Is there any data available on estimating how big an indexer's blockSignature database may become, based the blockSignSize and daily event volumes?

Tags (2)

jrodman
Splunk Employee
Splunk Employee

blockSIgnature events are just listings of eventIDs. Therefore a block signature event is just a bunch of numbers like 45:35 with spaces in between. Therefore the size of the block signature database has no correspondence with the size of the original events, making this difficult to estimate in bytes.

However, each event requires a price of around 8 bytes in the text, precompressed, and each event requires a hash and indexing-related overhead. I would guesstimate that each block, in in the current rawdata format and tsidx format, is going to cost on the order of a kilobyte. Empirical data is obviously vastly superior to my guesses.

There's therefore the obvious tunable here which is the blockSignSize. I think anyone using this feature significantly probably wants to tune that value upwards. You don't want it too high for low incoming datarates, but at higher rates, plausibly values like 10000 might be reasonable.

0 Karma

Michael_Wilde
Splunk Employee
Splunk Employee

Good question. Mine is the same size of my main index. Not thrilled.

Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

From Raw Data to Executive-Ready Stories, Faster

Build Data Stories for Every Audience  A dashboard is rarely just a dashboard. It might be the view an ...

Guided Onboarding with Auto-schema Is Now Generally Available

  We are excited to announce the General Availability of Guided Onboarding with Auto-Schematization ...

ATTENTION: We’re Moving! (AGAIN!)

The Splunk Community Slack is undergoing a system migration to keep our workspace secure and ...