Should I create a large number of indexes?

[Gong1027]

Dear Splunk experts,

Just want to ask about the general upside/downside of creating a large number of indexes.

Thinking to create a Splunk index per application/service so we may end up with 3K to 5K indexes

But this would allow us to target <<inputs.conf>> based on application/service

Just not sure of the downside of that many indexes...

Appreciate your advice.

[richgalloway · Accepted solution]

That's a lot of indexes - perhaps too many.  Having thousands of indexes means having tens (or hundreds) of thousands of buckets, which makes for a lot of files to open (subject to OS limits), decompress, and read.  It increases the chances of having lots of little indexes (and buckets) that are more metadata than data, wasting resources.

Splunk recommends putting data that is commonly used together in searches into the same index for more efficient searching.

There are a few reasons for creating a new index:

1) Data has different access requirements/restrictions

2) Data has different retention requirements

3) Data is of such volume that it warrants a separate index.

[Gong1027]

Thanks!

[isoutamo]

Hi

it’s just like @richgalloway said. Try to avoid to create any unnecessary indexes. There is upper limit of indexes both technical and usability point of view.

I assume that you have big indexer clusters in use and there are limit for max amount of buckets you could use some tenth millions I assume. I haven’t seen those limit since version 8 (in some conf presentation).

If you really need that amount of indexes then you probably must create several indexer clusters to manage that amount of buckets. In that case I suggest you to contact your local splunk partner or Splunk’s PS service to update your architecture!

r. Ismo

[Gong1027]

Thanks