topic Re: Creating large, multi-terabyte indexes in Getting Data In

Creating large, multi-terabyte indexes

williamsweat — Tue, 28 Feb 2012 18:38:35 GMT

Hi,

I have a few indexes that I want to expand to be multiple terabytes. Are there general guidelines about this? Should I increase the number of buckets, and if so what's considered 'just right' for a 2TB (or more) index?

What can I expect if I need to run an fsck? Will large indexes make running this out of the question?

Thanks,
Will

Re: Creating large, multi-terabyte indexes

wdhathaway — Tue, 28 Feb 2012 19:23:55 GMT

I'm not sure on the bucket size part of your question, but as far as your fsck question goes,
In general, fsck times are linear with number of inodes, so for a file system filled with a smaller number of large files (like Splunk indexes), it should be much faster to fsck than a file system filled with with a huge amount of small files.

Re: Creating large, multi-terabyte indexes

lguinn2 — Fri, 09 Mar 2012 02:20:21 GMT

Splunk automatically creates buckets as needed. You don't need to do anything about buckets for a 2TB index; this is not considered a particularly large index in Splunk. (There are customers who add much more than 2TB every day.)

However, you do need to change the maximum size of your index, as the default maximum size is 500,000MB (or .5TB) You can change this setting in the configuation file indexes.conf (maxTotalDataSizeMB) or you can do it via the user interface in the Splunk Manager.

Contrary to wdhathaway's post - a Splunk index is not implemented as a monolithic file; it is in fact a number of files. But I don't think that you will have a significant fsck problem anyway.

FInally, for more about indexes and sizing take a look at

Managing Indexes

Create and edit indexes