Knowledge Management

effectivity of multiple summary searches in a single index

bkoutsky
Engager

Hello,

I just discovered summary indexes (Oh joy! I can have results immediately instead of waiting a few minutes) and I'm starting to use them for a lot of my searches.

The question is: should I create multiple indexes, one for each summary search, or should I put all of them into one index? How efficient is Splunk at picking up summary search events from index using 'index=summary search_name="trololo"' ? "Everything in one index" is easier to use and maintain, but I want to avoid hitting a performance wall again after I accumulate some data.

So, for a lots of data and tens of summary searches, is it significantly faster to do

index='special-index-for-search-foo'

than

index='common-index' search_name='foo'

?
cheers,
Bob

Tags (1)
0 Karma

_d_
Splunk Employee
Splunk Employee

HI Bkoutsky,

Here is my recommendation: Yes. create multiple summary indexes but don't over do it.

I recommend creating a summary index for each search granularity. So, for example, I would have 3 summary indexes. The FIRST would be for summary searches that are less than an hour (5m, 10m, 15m, etc). I would use a SECOND one for hourly summaries up to a day and a THIRD one for summaries that are a day or longer.

I would label them like so: summary_5m, summary_1h and summary_1d

Having three summaries will allow you to set different retention policies and different rights/privileges for each, if necessary.

You can also do other cool things such as build the hourly summary index from the 5m summary and the daily summary from the hourly one!

Hope this helps!

d.

- please vote if you find this answer useful!

AlexMcDuffMille
Communicator

Can you expand on what you mean by 'don't over do it'.

Is there a tipping point in which Splunk starts to be a lot slower? Or is it a linear curve (2 searches will be half as fast as 1)

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!