topic Re: Significant search performance hit using multiple indexes in Splunk Search

Significant search performance hit using multiple indexes

andywins — Mon, 05 Aug 2013 17:34:02 GMT

I'm seeing three seconds of latency introduced to each search when using ~3,500 indexes. Here's the scenario:

~3,000,000 events for source "X", all stored within the main default index
No data in any of the other custom ~3,500 indexes
Free license
32g/ram, i7 series 4, 1tb/ssd
I've not violated the free tier indexing threshold

Using the admin account, a search as simple as "sourcetype=x" seems to wait roughly three seconds before beginning to fetch results. When I remove all the custom indexes, things fly like normal. Considering I'm not storing data in the extra indexes yet, I wouldn't expect a noticeable performance impact. As I create additional indexes, performance seems to drop.

What could be causing this? This link suggests it's not the number of indexes that count, it's the data. My experience shows the inverse. What is the maximum number of indexes per instance before running into issues?

Re: Significant search performance hit using multiple indexes

linu1988 — Mon, 05 Aug 2013 17:57:25 GMT

I wouldn't do this many index but curious to know the reason as i see something similar with me.

Re: Significant search performance hit using multiple indexes

alacercogitatus — Mon, 05 Aug 2013 18:17:35 GMT

I'd wager that it is still tied to the number of indexes. Even if those indexes don't contain data, each bloomfilter on each index bucket must be checked for matches. So even though its a quick search of the bloomfilter, you are still performing it ~3500 * #BucketsInIndex times. Try doing your search like this and see if it speeds up:

index=main sourcetype=x| blah blah

http://docs.splunk.com/Splexicon:Bloomfilter

Re: Significant search performance hit using multiple indexes

linu1988 — Mon, 05 Aug 2013 18:42:01 GMT

But when i added another 20 indexes to existing 20 i felt the searches had slowed down even if the new indexes didn't contain any data. Any possible reason?

Re: Significant search performance hit using multiple indexes

alacercogitatus — Mon, 05 Aug 2013 19:11:12 GMT

But if you don't explicitly declare in your search which index, the search still hits every index location, and depending on your disk speeds, might introduce some latency.

Re: Significant search performance hit using multiple indexes

linu1988 — Mon, 05 Aug 2013 19:42:35 GMT

Sorry i should have been specific, I meant the dashboards which are having proper search queries.

Re: Significant search performance hit using multiple indexes

gkanapathy — Mon, 05 Aug 2013 19:54:28 GMT

Uh, 3500 indexes is way above the expected parameters. I would hesitate to use more than 200. Frankly, I would be happy to only see 3 seconds of latency on searches with that many indexes, but I suspect that's because you have a good SSD. Every index of course means more places to look for every search (even if it's empty, it's impossible to know it's empty w/o looking) as well as overhead checking for whether its full, etc.

What are you using 3500 indexes for? I wonder whether you need that or whether you can just put them into fewer.

Re: Significant search performance hit using multiple indexes

andywins — Tue, 06 Aug 2013 13:54:30 GMT

Thanks for the comments. I'll be bulk loading some data this week to vet the suggested approach. I'm using two Samsung 840 pro 500gb ssd's, striped. gkanapathy, we have this many customers and I want to partition their data, both for speed and security. Right now I can create roles/users locked down to a specific index. Ideally, I would have an index for each datafeed per customer (15 feeds * 3500 clients = 52,500 indexes). Let me know if you can think of a better approach.

Re: Significant search performance hit using multiple indexes

Ayn — Tue, 06 Aug 2013 13:57:40 GMT

As long as you can single out the indexes you need for each query this won't be as big a problem since in that case Splunk knows immediately which indexes to open. If you do need to run loads and loads of searches over ALL indexes that could be more problematic.

Re: Significant search performance hit using multiple indexes

emiller42 — Tue, 06 Aug 2013 15:06:27 GMT

I would recommend one index per customer. You can use sourcetypes to differentiate the datafeeds.

Re: Significant search performance hit using multiple indexes

alacercogitatus — Tue, 06 Aug 2013 16:10:54 GMT

Just a reminder: please accept the answer if we have answered your question. Thanks!

Re: Significant search performance hit using multiple indexes

andywins — Fri, 16 Aug 2013 17:51:55 GMT

Indeed, limiting to explicit indexes increases performance. Thanks alacercogitatus