Why am I getting less search results when reading data from the KV store via inputlookup?

[szabados]

I have a KV store defined on my search head, which is not replicated.
I'm populating it with data with a scheduled search (outputlookup), which is producing approx 750.000 rows.
When I'm reading data from the KV store via inputlookup, I'm getting only like 1500 rows.

The populating search takes really long time to finish, because of the |outputlookup my_kvstore command at the end of the search.
If I remove this and replace it with an outputcsv, it is much faster.

What are the limitations of KV stores, and how can I troubleshoot this?
I was looking at the mongod.log, but haven't seen any errors.

[muebel]

Hi szabados,

Your config seems to be reasonable, and so based on your comments I'd suggest trying to reinitialize this particular kvstore.

Alternatively, you could try an intermediary csv as a troubleshooting step/workaround. That is, first output the generating search to a csv file, and then use inputlookup to read that csv, and then output to kvstore.

There shouldn't be any issue with a generating search taking a long time, and therefore not correctly writing results to a kvstore, but inputlookup is fast, and so we can hopefully rule that out.

Also, if this is still an issue, can we see what the generating search looks like?

Please let me know if this answers your question! 😄

[szabados]

It is defined with 13 fields in my collections.conf, all strings like this:

field.field1 = string
field.field2 = string
[...]
accelerated_fields.my_accel = {"field1"}
replicate = false

I refer to it in my transforms.conf like this:

collection = mycollection
external_type = kvstore
fields_list = _key, field1, field2, ...

When the outputlookup search is run, I'm not using explicitly any _key fields, but before that command, the results are narrowed down with the fields command to have only these fields which are used in the KV store as well.

Other KV store lookups, which are defined and used the same way seems to be working, and recently I've had an error where the KV store was unable to initialize, so I'm suspecting this issue is rather with the KV store subsystem itself, not this particular lookup.
I've been searching for that error as well, but the only answer I've found here is about the expired SSL certificate, but that does not apply to my case.

[szabados]

It is defined with 13 fields in my collections.conf, all strings like this:

field.field1 = string
field.field2 = string
[...]
accelerated_fields.my_accel = {"field1"}
replicate = false

I refer to it in my transforms.conf like this:

collection = mycollection
external_type = kvstore
fields_list = _key, field1, field2, ...

When the outputlookup search is run, I'm not using explicitly any _key fields, but before that command, the results are narrowed down with the fields command to have only these fields which are used in the KV store as well.

Other KV store lookups, which are defined and used the same way seems to be working, and recently I've had an error where the KV store was unable to initialize, so I'm suspecting this issue is rather with the KV store subsystem itself, not this particular lookup.
I've been searching for that error as well, but the only answer I've found here is about the expired SSL certificate, but that does not apply to my case.

[muebel]

whats the collections.conf and transforms.conf config that defines the lookup?

whats the resulting field set passed to outputlookup? Do you use table or something similar to filter it down to the specific fields specified in the kvstore? What's the _key field?

[szabados]

Please see my answer below, I wanted to post it here, but overlooked where I'm typing sorry.