Splunk Search

Why is Automated lookup using kvstore collection not working?

wmuselle
Explorer

I have created a collection in app/local/collections.conf

a matching lookup in app/local/transforms.conf

I have 5 key fields which together for the unique key, the combination of these is also stored in the _key field.

The data is populated from an index which is filled from a dbconnect source, and automatically updated up into to collection. All this works just fine.

when I use the lookup in SPL using the five fields as input, I nicely get referenced data back. when I create this lookup as part of a data model, it also provides the extra fields in the datamodel.

However if I try to use this in an automated lookup, I cannot get it to work.

I have verified the correct use of the sourcetype (and also tried defining against source)

I have verified the rights and tried using all on app and global level

I have duplicated the full config on a csv file and this works just fine

but against the kvstore the automatic lookup just wont work.

illustration of the files and configs

 

 

 

 

 

 

collections.conf  in app/local
[my_collection]
field.inputfield1 = string
field.inputfield2 = string
field.inputfield3 = string
field.inputfield4 = string
field.inputfield5 = string
field.outputfield1 = string
...

 

 

 

 

 

 

 

 

 

 

 

 

 

transforms.conf  in app/local
[my_collection_lookup]
external_type = kvstore
collection = my_collection
fields_list = _key, inputfield1, inputfield2,inputfield3,inputfield4,inputfield5, outputfield1 ...

 

 

 

 

 

 

 

 

 

 

 

 

 

props.conf in app/local
[sourcetype_stanza]
LOOKUP-enrich_kv = my_collection_lookup inputfield1 AS datafield1 inputfield2 AS datafield2 inputfield3 AS datafield3 inputfield4 AS datafield4 inputfield5 as datafield5 OUTPUTNEW _key as key outputfield1 ....

 

 

 

 

 

 

 

 

any experiences/thoughts/ideas ?

Labels (1)
0 Karma
1 Solution

wmuselle
Explorer

found it for reference :

https://docs.splunk.com/Documentation/Splunk/8.1.3/Knowledge/Makeyourlookupautomatic 

Enable replication for a KV store collection
In Splunk Enterprise, KV Store collections are not bundle-replicated to indexers by default, and lookups run locally on the search head rather than on remote peers. When you enable replication for a KV Store collection, you can run the lookups on your indexers which let you use automatic lookups with your KV Store collections.

To enable replication for a KV Store collection and allow lookups against that collection to be automatic:

  • Open collections.conf.
  • Set replicate to true in the stanza for the collection.

This parameter is set to false by default.
Restart Splunk Enterprise to apply your changes.

 

View solution in original post

wmuselle
Explorer

found it for reference :

https://docs.splunk.com/Documentation/Splunk/8.1.3/Knowledge/Makeyourlookupautomatic 

Enable replication for a KV store collection
In Splunk Enterprise, KV Store collections are not bundle-replicated to indexers by default, and lookups run locally on the search head rather than on remote peers. When you enable replication for a KV Store collection, you can run the lookups on your indexers which let you use automatic lookups with your KV Store collections.

To enable replication for a KV Store collection and allow lookups against that collection to be automatic:

  • Open collections.conf.
  • Set replicate to true in the stanza for the collection.

This parameter is set to false by default.
Restart Splunk Enterprise to apply your changes.

 

DmitriyGolovnya
Engager

Hi! may be you know why after adding replicate = true to KV lookup in collections.conf, my datamodel’s lookup isn’t working, even in datamodel editor (preview). it just can’t join, all output fields is empty (without replicate all works fine)?

0 Karma
Get Updates on the Splunk Community!

BSides Splunk 2022 - The Call for Papers is now Open!

TLDR; Main Site: https://bsidessplunk.com CFP Site: https://bsidessplunk.com/cfp CFP Opens: December 15th, ...

Sending Metrics to Splunk Enterprise With the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

What's New in Splunk Cloud Platform 9.0.2208?!

Howdy!  We are happy to share the newest updates in Splunk Cloud Platform 9.0.2208! Analysts can benefit ...