Splunk Search

Automated lookup using kvstore collection not working

wmuselle
Explorer

I have created a collection in app/local/collections.conf

a matching lookup in app/local/transforms.conf

I have 5 key fields which together for the unique key, the combination of these is also stored in the _key field.

The data is populated from an index which is filled from a dbconnect source, and automatically updated up into to collection. All this works just fine.

when I use the lookup in SPL using the five fields as input, I nicely get referenced data back. when I create this lookup as part of a data model, it also provides the extra fields in the datamodel.

However if I try to use this in an automated lookup, I cannot get it to work.

I have verified the correct use of the sourcetype (and also tried defining against source)

I have verified the rights and tried using all on app and global level

I have duplicated the full config on a csv file and this works just fine

but against the kvstore the automatic lookup just wont work.

illustration of the files and configs

 

 

 

 

collections.conf  in app/local
[my_collection]
field.inputfield1 = string
field.inputfield2 = string
field.inputfield3 = string
field.inputfield4 = string
field.inputfield5 = string
field.outputfield1 = string
...

 

 

 

 

 

 

 

 

 

transforms.conf  in app/local
[my_collection_lookup]
external_type = kvstore
collection = my_collection
fields_list = _key, inputfield1, inputfield2,inputfield3,inputfield4,inputfield5, outputfield1 ...

 

 

 

 

 

 

 

 

 

props.conf in app/local
[sourcetype_stanza]
LOOKUP-enrich_kv = my_collection_lookup inputfield1 AS datafield1 inputfield2 AS datafield2 inputfield3 AS datafield3 inputfield4 AS datafield4 inputfield5 as datafield5 OUTPUTNEW _key as key outputfield1 ....

 

 

 

 

 

 

any experiences/thoughts/ideas ?

Labels (1)
0 Karma
1 Solution

wmuselle
Explorer

found it for reference :

https://docs.splunk.com/Documentation/Splunk/8.1.3/Knowledge/Makeyourlookupautomatic 

Enable replication for a KV store collection
In Splunk Enterprise, KV Store collections are not bundle-replicated to indexers by default, and lookups run locally on the search head rather than on remote peers. When you enable replication for a KV Store collection, you can run the lookups on your indexers which let you use automatic lookups with your KV Store collections.

To enable replication for a KV Store collection and allow lookups against that collection to be automatic:

  • Open collections.conf.
  • Set replicate to true in the stanza for the collection.

This parameter is set to false by default.
Restart Splunk Enterprise to apply your changes.

 

View solution in original post

wmuselle
Explorer

found it for reference :

https://docs.splunk.com/Documentation/Splunk/8.1.3/Knowledge/Makeyourlookupautomatic 

Enable replication for a KV store collection
In Splunk Enterprise, KV Store collections are not bundle-replicated to indexers by default, and lookups run locally on the search head rather than on remote peers. When you enable replication for a KV Store collection, you can run the lookups on your indexers which let you use automatic lookups with your KV Store collections.

To enable replication for a KV Store collection and allow lookups against that collection to be automatic:

  • Open collections.conf.
  • Set replicate to true in the stanza for the collection.

This parameter is set to false by default.
Restart Splunk Enterprise to apply your changes.

 

View solution in original post

Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!