Splunk Search

Automated lookup using KVstore lookup

coreyCLI
Path Finder

I have a KV store collection that is populated.  I have a lookup definition pointing to the KV store.  If you use the kvstore lookup definition in a search, I get matching results and everything works as expected.  

 

index=* source=jello
| lookup kvstore_lookup ip as srcip outputnew city as src_city

 

However, if I move that into an automatic lookup it does not work. 

Before using the kvstore I was using a csv lookup and the automatic lookups where working fine.  The csv grew to 122mb so I populated a kvstore with the below.  

 

| inputlookup old_csv_lookup 
| outputlookup kvstore_lookup

 

Permissions on the automatic lookups are global, everyone read, admin write.  I can see in the search log that its calling the automatic lookup "Will use Lookup: Lookup-......" but the the fields that are supposed to be added in from the lookup dont populate.

Also, I am using matchtype=CIDR for this lookup definition.

Any ideas why the automatic lookup is not working now that its using the kvstore? 

Labels (1)
Tags (1)
0 Karma

coreyCLI
Path Finder

For anyone tracking this. If you  migrate to wiredTiger you will loose the metrics for "Accelerations" and "Accelerated Size (MB)".  If you want to fix this you can add these regexes to the existing search in the DMC-->"KV Store Instance"-->"Collection Metrics" panel

| rex field=data "nindexes\"\:(?<nindexes>\d+)\,"

| rex field=data "totalIndexeSize\"\:(?<totalIndexSize>\d+)\,"

 

0 Karma

wmuselle
Explorer

actually yes:

found it for reference :

https://docs.splunk.com/Documentation/Splunk/8.1.3/Knowledge/Makeyourlookupautomatic 

Enable replication for a KV store collection
In Splunk Enterprise, KV Store collections are not bundle-replicated to indexers by default, and lookups run locally on the search head rather than on remote peers. When you enable replication for a KV Store collection, you can run the lookups on your indexers which let you use automatic lookups with your KV Store collections.

To enable replication for a KV Store collection and allow lookups against that collection to be automatic:

  1. Open collections.conf.
    Set replicate to true in the stanza for the collection.
  2. This parameter is set to false by default.
    Restart Splunk Enterprise to apply your changes.

0 Karma

coreyCLI
Path Finder

That's interesting.  Not sure how I haven't come across that document before.  We must have some other issues on this particular instance because replicating the KV store to the indexers did not help.  Also, when you look at the kvstore pages in the DMC it doesn't show the accelerated fields status either.  

Thanks for the info!

0 Karma

wmuselle
Explorer

I have run into exactly this issue and was going to post on it.

symptoms: like above, exact replica configuration using csv works just fine.

executing the lookup piped in spl works just fine

defining the same lookup on a data model works just fine.

 

just the automatic lookup doesnt, I have tried both on sourcetype and source

0 Karma

coreyCLI
Path Finder

In a way I am glad to hear someone else is having this issue! lol.  Have you found any solutions?  Possible bug?

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!