Splunk Search

Why is Automated lookup using kvstore collection not working?

wmuselle
Explorer

I have created a collection in app/local/collections.conf

a matching lookup in app/local/transforms.conf

I have 5 key fields which together for the unique key, the combination of these is also stored in the _key field.

The data is populated from an index which is filled from a dbconnect source, and automatically updated up into to collection. All this works just fine.

when I use the lookup in SPL using the five fields as input, I nicely get referenced data back. when I create this lookup as part of a data model, it also provides the extra fields in the datamodel.

However if I try to use this in an automated lookup, I cannot get it to work.

I have verified the correct use of the sourcetype (and also tried defining against source)

I have verified the rights and tried using all on app and global level

I have duplicated the full config on a csv file and this works just fine

but against the kvstore the automatic lookup just wont work.

illustration of the files and configs

 

 

 

 

 

 

collections.conf  in app/local
[my_collection]
field.inputfield1 = string
field.inputfield2 = string
field.inputfield3 = string
field.inputfield4 = string
field.inputfield5 = string
field.outputfield1 = string
...

 

 

 

 

 

 

 

 

 

 

 

 

 

transforms.conf  in app/local
[my_collection_lookup]
external_type = kvstore
collection = my_collection
fields_list = _key, inputfield1, inputfield2,inputfield3,inputfield4,inputfield5, outputfield1 ...

 

 

 

 

 

 

 

 

 

 

 

 

 

props.conf in app/local
[sourcetype_stanza]
LOOKUP-enrich_kv = my_collection_lookup inputfield1 AS datafield1 inputfield2 AS datafield2 inputfield3 AS datafield3 inputfield4 AS datafield4 inputfield5 as datafield5 OUTPUTNEW _key as key outputfield1 ....

 

 

 

 

 

 

 

 

any experiences/thoughts/ideas ?

Labels (1)
0 Karma
1 Solution

wmuselle
Explorer

found it for reference :

https://docs.splunk.com/Documentation/Splunk/8.1.3/Knowledge/Makeyourlookupautomatic 

Enable replication for a KV store collection
In Splunk Enterprise, KV Store collections are not bundle-replicated to indexers by default, and lookups run locally on the search head rather than on remote peers. When you enable replication for a KV Store collection, you can run the lookups on your indexers which let you use automatic lookups with your KV Store collections.

To enable replication for a KV Store collection and allow lookups against that collection to be automatic:

  • Open collections.conf.
  • Set replicate to true in the stanza for the collection.

This parameter is set to false by default.
Restart Splunk Enterprise to apply your changes.

 

View solution in original post

wmuselle
Explorer

found it for reference :

https://docs.splunk.com/Documentation/Splunk/8.1.3/Knowledge/Makeyourlookupautomatic 

Enable replication for a KV store collection
In Splunk Enterprise, KV Store collections are not bundle-replicated to indexers by default, and lookups run locally on the search head rather than on remote peers. When you enable replication for a KV Store collection, you can run the lookups on your indexers which let you use automatic lookups with your KV Store collections.

To enable replication for a KV Store collection and allow lookups against that collection to be automatic:

  • Open collections.conf.
  • Set replicate to true in the stanza for the collection.

This parameter is set to false by default.
Restart Splunk Enterprise to apply your changes.

 

DmitriyGolovnya
Engager

Hi! may be you know why after adding replicate = true to KV lookup in collections.conf, my datamodel’s lookup isn’t working, even in datamodel editor (preview). it just can’t join, all output fields is empty (without replicate all works fine)?

0 Karma
Get Updates on the Splunk Community!

Introducing Ingest Actions: Filter, Mask, Route, Repeat

WATCH NOW Ingest Actions (IA) is the best new way to easily filter, mask and route your data in Splunk® ...

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...