Splunk Search

Why is Automated lookup using kvstore collection not working?

wmuselle
Path Finder

I have created a collection in app/local/collections.conf

a matching lookup in app/local/transforms.conf

I have 5 key fields which together for the unique key, the combination of these is also stored in the _key field.

The data is populated from an index which is filled from a dbconnect source, and automatically updated up into to collection. All this works just fine.

when I use the lookup in SPL using the five fields as input, I nicely get referenced data back. when I create this lookup as part of a data model, it also provides the extra fields in the datamodel.

However if I try to use this in an automated lookup, I cannot get it to work.

I have verified the correct use of the sourcetype (and also tried defining against source)

I have verified the rights and tried using all on app and global level

I have duplicated the full config on a csv file and this works just fine

but against the kvstore the automatic lookup just wont work.

illustration of the files and configs

 

 

 

 

 

 

collections.conf  in app/local
[my_collection]
field.inputfield1 = string
field.inputfield2 = string
field.inputfield3 = string
field.inputfield4 = string
field.inputfield5 = string
field.outputfield1 = string
...

 

 

 

 

 

 

 

 

 

 

 

 

 

transforms.conf  in app/local
[my_collection_lookup]
external_type = kvstore
collection = my_collection
fields_list = _key, inputfield1, inputfield2,inputfield3,inputfield4,inputfield5, outputfield1 ...

 

 

 

 

 

 

 

 

 

 

 

 

 

props.conf in app/local
[sourcetype_stanza]
LOOKUP-enrich_kv = my_collection_lookup inputfield1 AS datafield1 inputfield2 AS datafield2 inputfield3 AS datafield3 inputfield4 AS datafield4 inputfield5 as datafield5 OUTPUTNEW _key as key outputfield1 ....

 

 

 

 

 

 

 

 

any experiences/thoughts/ideas ?

Labels (1)
0 Karma
1 Solution

wmuselle
Path Finder

found it for reference :

https://docs.splunk.com/Documentation/Splunk/8.1.3/Knowledge/Makeyourlookupautomatic 

Enable replication for a KV store collection
In Splunk Enterprise, KV Store collections are not bundle-replicated to indexers by default, and lookups run locally on the search head rather than on remote peers. When you enable replication for a KV Store collection, you can run the lookups on your indexers which let you use automatic lookups with your KV Store collections.

To enable replication for a KV Store collection and allow lookups against that collection to be automatic:

  • Open collections.conf.
  • Set replicate to true in the stanza for the collection.

This parameter is set to false by default.
Restart Splunk Enterprise to apply your changes.

 

View solution in original post

wmuselle
Path Finder

found it for reference :

https://docs.splunk.com/Documentation/Splunk/8.1.3/Knowledge/Makeyourlookupautomatic 

Enable replication for a KV store collection
In Splunk Enterprise, KV Store collections are not bundle-replicated to indexers by default, and lookups run locally on the search head rather than on remote peers. When you enable replication for a KV Store collection, you can run the lookups on your indexers which let you use automatic lookups with your KV Store collections.

To enable replication for a KV Store collection and allow lookups against that collection to be automatic:

  • Open collections.conf.
  • Set replicate to true in the stanza for the collection.

This parameter is set to false by default.
Restart Splunk Enterprise to apply your changes.

 

DmitriyGolovnya
Engager

Hi! may be you know why after adding replicate = true to KV lookup in collections.conf, my datamodel’s lookup isn’t working, even in datamodel editor (preview). it just can’t join, all output fields is empty (without replicate all works fine)?

0 Karma
Get Updates on the Splunk Community!

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...

Announcing the public preview of Splunk Data Management Ingest Processor

We are pleased to announce the public preview of Splunk’s latest data management capability - Ingest Processor ...

Splunk Lantern is up for three awards! Can we get your vote?

It’s been a big news month for Splunk Lantern. Along with celebrating the publication of our 1000th article, ...