Getting Data In

Creating large, multi-terabyte indexes

williamsweat
Path Finder

Hi,

I have a few indexes that I want to expand to be multiple terabytes. Are there general guidelines about this? Should I increase the number of buckets, and if so what's considered 'just right' for a 2TB (or more) index?

What can I expect if I need to run an fsck? Will large indexes make running this out of the question?

Thanks,
Will

Tags (1)
0 Karma

lguinn2
Legend

Splunk automatically creates buckets as needed. You don't need to do anything about buckets for a 2TB index; this is not considered a particularly large index in Splunk. (There are customers who add much more than 2TB every day.)

However, you do need to change the maximum size of your index, as the default maximum size is 500,000MB (or .5TB) You can change this setting in the configuation file indexes.conf (maxTotalDataSizeMB) or you can do it via the user interface in the Splunk Manager.

Contrary to wdhathaway's post - a Splunk index is not implemented as a monolithic file; it is in fact a number of files. But I don't think that you will have a significant fsck problem anyway.

FInally, for more about indexes and sizing take a look at

Managing Indexes

Create and edit indexes

wdhathaway
Explorer

I'm not sure on the bucket size part of your question, but as far as your fsck question goes,
In general, fsck times are linear with number of inodes, so for a file system filled with a smaller number of large files (like Splunk indexes), it should be much faster to fsck than a file system filled with with a huge amount of small files.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...