topic Re: Automated lookup using KVstore lookup in Splunk Search

Automated lookup using KVstore lookup

coreyCLI — Thu, 25 Mar 2021 12:46:00 GMT

I have a KV store collection that is populated. I have a lookup definition pointing to the KV store. If you use the kvstore lookup definition in a search, I get matching results and everything works as expected.

index=* source=jello | lookup kvstore_lookup ip as srcip outputnew city as src_city

However, if I move that into an automatic lookup it does not work.

Before using the kvstore I was using a csv lookup and the automatic lookups where working fine. The csv grew to 122mb so I populated a kvstore with the below.

| inputlookup old_csv_lookup | outputlookup kvstore_lookup

Permissions on the automatic lookups are global, everyone read, admin write. I can see in the search log that its calling the automatic lookup "Will use Lookup: Lookup-......" but the the fields that are supposed to be added in from the lookup dont populate.

Also, I am using matchtype=CIDR for this lookup definition.

Any ideas why the automatic lookup is not working now that its using the kvstore?

Re: Automated lookup using KVstore lookup

wmuselle — Thu, 06 May 2021 13:00:49 GMT

I have run into exactly this issue and was going to post on it.

symptoms: like above, exact replica configuration using csv works just fine.

executing the lookup piped in spl works just fine

defining the same lookup on a data model works just fine.

just the automatic lookup doesnt, I have tried both on sourcetype and source

Re: Automated lookup using KVstore lookup

coreyCLI — Tue, 18 May 2021 10:44:29 GMT

In a way I am glad to hear someone else is having this issue! lol. Have you found any solutions? Possible bug?

Re: Automated lookup using KVstore lookup

wmuselle — Thu, 20 May 2021 10:33:20 GMT

actually yes:

found it for reference :

https://docs.splunk.com/Documentation/Splunk/8.1.3/Knowledge/Makeyourlookupautomatic

Enable replication for a KV store collection
In Splunk Enterprise, KV Store collections are not bundle-replicated to indexers by default, and lookups run locally on the search head rather than on remote peers. When you enable replication for a KV Store collection, you can run the lookups on your indexers which let you use automatic lookups with your KV Store collections.

To enable replication for a KV Store collection and allow lookups against that collection to be automatic:

Open collections.conf.
Set replicate to true in the stanza for the collection.
This parameter is set to false by default.
Restart Splunk Enterprise to apply your changes.

Re: Automated lookup using KVstore lookup

coreyCLI — Thu, 20 May 2021 10:44:47 GMT

That's interesting. Not sure how I haven't come across that document before. We must have some other issues on this particular instance because replicating the KV store to the indexers did not help. Also, when you look at the kvstore pages in the DMC it doesn't show the accelerated fields status either.

Thanks for the info!

Re: Automated lookup using KVstore lookup

coreyCLI — Wed, 07 Jul 2021 13:00:14 GMT

For anyone tracking this. If you migrate to wiredTiger you will loose the metrics for "Accelerations" and "Accelerated Size (MB)". If you want to fix this you can add these regexes to the existing search in the DMC-->"KV Store Instance"-->"Collection Metrics" panel

| rex field=data "nindexes\"\:(?<nindexes>\d+)\," | rex field=data "totalIndexeSize\"\:(?<totalIndexSize>\d+)\,"

Re: Automated lookup using KVstore lookup

Pony0 — Thu, 09 Nov 2023 12:58:37 GMT

Hi !

I am facing a very similar issue : after adding a new field to my KV store automatic lookup doesn't work and never returns my new field in my events but I can manually retrieve it with this query :

| inputlookup my_kvstore

but that one :

index=my_index | lookup my_kvstore...

throws an error :

[comma separated of my indexers list] phase_0 - Streamed search execute failed because: Error in 'lookup' command: Cannot find the destination field 'my_new_field' in the lookup table 'my_kvstore'..

still, with this query :

index=my_index | lookup local=true my_kvstore...

I can retrieve my new field...

Regards,