topic Re: Why does Splunk have multiple indexes? in Getting Data In

Why does Splunk have multiple indexes?

maverick — Thu, 22 Apr 2010 21:23:28 GMT

With Splunk's normalizing timestamp-based event indexing capabilities combined with it's powerful search language and processing commands, one would think that all you need is one big main index.

So why is there more than one index and what are the reasons for creating additional indexes?

Re: Why does Splunk have multiple indexes?

gkanapathy — Thu, 22 Apr 2010 22:12:49 GMT

Mulitple indexes are indicated usually for two reasons:

Physical data separation
- This may be related to access control of data, but it is not necessary to use separate indexes to control access to data, although with current (v4.1) Splunk management capabilities, access control is easiest to configure with separate indexes.
Differential retention periods for different data sets
- This includes summary indexing of different time densities, test indexes, as well as cases of some data having longer retention requirements than other (often extremely high-volume) data has shorter requirements.

Performance is not a typical consideration, and the effect of multiple indexes vs a single one for a given set of data varies greatly depending on the exact nature of the data and the exact queries or mix of queries to be performed against it.

Re: Why does Splunk have multiple indexes?

muebel — Thu, 22 Apr 2010 22:44:40 GMT

In addition to gkanapathy's answer, additional indexes seems to be part and parcel of how summary indexing works.

http://www.splunk.com/base/Documentation/4.1.1/Knowledge/Usesummaryindexing

Re: Why does Splunk have multiple indexes?

jrodman — Fri, 23 Apr 2010 02:37:22 GMT

There are performance goals as well, sparse data (login errors) will be more performant when searched apart from bulk data (firewall rule traversals). There's administrative overhead in creating multiple indexes (you have to configure them) but when you will have a large amount of data of quite different volumes in high performance environments this can be worthwhile. This is the main reason that summary indexing goes to a new index (it could use the same one).

There are more obscure cases as well for performance, such as different segmentation per index, but ideally this is not necessary.

Re: Why does Splunk have multiple indexes?

muebel — Fri, 23 Apr 2010 02:40:38 GMT

I have no idea why this was considered the best answer hah

Re: Why does Splunk have multiple indexes?

gkanapathy — Sun, 25 Apr 2010 04:31:06 GMT

maverick is on vendetta against me, jrodman, and other Splunk employees on this site.

Re: Why does Splunk have multiple indexes?

Genti — Thu, 07 Oct 2010 00:30:05 GMT

ha! bad Maverick, bad!