topic Re: Automated lookup using kvstore collection not working in Splunk Search

Why is Automated lookup using kvstore collection not working?

wmuselle — Mon, 06 Jun 2022 20:52:59 GMT

I have created a collection in app/local/collections.conf

a matching lookup in app/local/transforms.conf

I have 5 key fields which together for the unique key, the combination of these is also stored in the _key field.

The data is populated from an index which is filled from a dbconnect source, and automatically updated up into to collection. All this works just fine.

when I use the lookup in SPL using the five fields as input, I nicely get referenced data back. when I create this lookup as part of a data model, it also provides the extra fields in the datamodel.

However if I try to use this in an automated lookup, I cannot get it to work.

I have verified the correct use of the sourcetype (and also tried defining against source)

I have verified the rights and tried using all on app and global level

I have duplicated the full config on a csv file and this works just fine

but against the kvstore the automatic lookup just wont work.

illustration of the files and configs

collections.conf in app/local [my_collection] field.inputfield1 = string field.inputfield2 = string field.inputfield3 = string field.inputfield4 = string field.inputfield5 = string field.outputfield1 = string ...

transforms.conf in app/local [my_collection_lookup] external_type = kvstore collection = my_collection fields_list = _key, inputfield1, inputfield2,inputfield3,inputfield4,inputfield5, outputfield1 ...

props.conf in app/local [sourcetype_stanza] LOOKUP-enrich_kv = my_collection_lookup inputfield1 AS datafield1 inputfield2 AS datafield2 inputfield3 AS datafield3 inputfield4 AS datafield4 inputfield5 as datafield5 OUTPUTNEW _key as key outputfield1 ....

any experiences/thoughts/ideas ?

Re: Automated lookup using kvstore collection not working

wmuselle — Thu, 20 May 2021 10:32:19 GMT

found it for reference :

https://docs.splunk.com/Documentation/Splunk/8.1.3/Knowledge/Makeyourlookupautomatic

Enable replication for a KV store collection
In Splunk Enterprise, KV Store collections are not bundle-replicated to indexers by default, and lookups run locally on the search head rather than on remote peers. When you enable replication for a KV Store collection, you can run the lookups on your indexers which let you use automatic lookups with your KV Store collections.

To enable replication for a KV Store collection and allow lookups against that collection to be automatic:

Open collections.conf.
Set replicate to true in the stanza for the collection.

This parameter is set to false by default.
Restart Splunk Enterprise to apply your changes.

Re: Automated lookup using kvstore collection not working

DmitriyGolovnya — Mon, 06 Jun 2022 11:55:23 GMT

Hi! may be you know why after adding replicate = true to KV lookup in collections.conf, my datamodel’s lookup isn’t working, even in datamodel editor (preview). it just can’t join, all output fields is empty (without replicate all works fine)?