Hi everyone!
We've moved some of heavy lookups to kv store and now they work faster and more stable. But one of them barely can calculate something, being inserted in a search with lookup even on 24 hour span, while it easily loads with inputlookup.
Accelerated field is a hexadecimal uid of 16 symbols. To make it easier for splunk to accelerate it, we converted it in a bigint and accelerated on this new field. Still the same slowness. It was rewritten but it didn't help. And its csv version lookups much faster, but we had some issues with updating large lookups so we moved to kv.
Slow-lookuped kv has 10mln entries and around 5 fields.
Fast-lookuped kv with 40mln entries and 10 fields works fine (with accelerated field also converted from hex to bigint)
Server: 251.81 GB Physical Memory, 20 CPU Cores
in limits.conf:
max_threads_per_outputlookup = 10
in server.conf:
oplogSize = 2000
What can cause such slowness? What params can we tune?
