Will there be performance issues using KV Store with a large data set?

nawneel — Tue, 29 Sep 2020 09:35:14 GMT

Hi all

I have a large data set (20 million) since 2015 which keeps on growing. In my case, I am supposed to use lookup and I found out that KV store is best since records in index are getting updated with _key(ORDER_KEY) remaining constant, hence my lookup will also be updating. Now with this huge set of growing data, will I land in to some sort of performance issue?

I thought of using multiple KV Store lookup broken down by month such as events from nov2015 will go to kvlookup_nov2015 and events from dec2015 will go to kvlookup_dec2015 based on ORDER_KEY creation time, all the collection and transforms.conf entries for lookup definitions will be made earlier only, but I am not able to achieve this as run time in search |outputlookup.

I tried the macro approach with eval based definition. |outputlookup `filename(ORDER_KEY)`.

[filename(1)]
args = ORDER_KEY
definition =(case(match($ORDER_KEY$, "^201511.*"),"csv_lookup_nov_2015",match($ORDER_KEY$,"^201512.*"),"csv_lookup_dec_2015"))

It did not work for me. Please help me out

Thanks in advance.

Re: Will there be performance issues using KV Store with a large data set?

richgalloway — Mon, 02 May 2016 12:07:25 GMT

KV Store is intended to be use for large data sets so I'd continue to use a single lookup rather than have to update your macro every month. If you have doubts, Google mongodb.

topic Will there be performance issues using KV Store with a large data set? in Monitoring Splunk

Will there be performance issues using KV Store with a large data set?

Re: Will there be performance issues using KV Store with a large data set?