Monitoring Splunk
Highlighted

Will there be performance issues using KV Store with a large data set?

Communicator

Hi all

I have a large data set (20 million) since 2015 which keeps on growing. In my case, I am supposed to use lookup and I found out that KV store is best since records in index are getting updated with key(ORDERKEY) remaining constant, hence my lookup will also be updating. Now with this huge set of growing data, will I land in to some sort of performance issue?

I thought of using multiple KV Store lookup broken down by month such as events from nov2015 will go to kvlookupnov2015 and events from dec2015 will go to kvlookupdec2015 based on ORDER_KEY creation time, all the collection and transforms.conf entries for lookup definitions will be made earlier only, but I am not able to achieve this as run time in search |outputlookup.

I tried the macro approach with eval based definition. |outputlookup `filename(ORDER_KEY)`.

[filename(1)]
args = ORDER_KEY
definition =(case(match($ORDER_KEY$, "^201511.*"),"csv_lookup_nov_2015",match($ORDER_KEY$,"^201512.*"),"csv_lookup_dec_2015"))

It did not work for me. Please help me out

Thanks in advance.

0 Karma
Highlighted

Re: Will there be performance issues using KV Store with a large data set?

SplunkTrust
SplunkTrust

KV Store is intended to be use for large data sets so I'd continue to use a single lookup rather than have to update your macro every month. If you have doubts, Google mongodb.

---
If this reply helps you, an upvote would be appreciated.
0 Karma