Archive

Search Head Cluster: Lookups definitions not replicated to indexers

Path Finder

I have a search head clusters with an indexer cluster, version 7.2.3.

On a search head, using Web UI I created a new file-based lookup in Search. On a search head I did a dummy search (which didn't involve the indexer) and made sure that the lookup is working. However, when I do a search which involves the indexer, the lookup failed.

On my indexer, I found that the lookup file was successfully replicated (I found it in a sub-folder under $SPLUNK_HOME/var/run/searchpeers/). I looked at the search.log in the indexer and saw that the indexer cannot find the lookup definition. So I think the lookup definition itself doesn't seem to be replicated.

Are lookup definitions replicated by default to the indexer? And if it's replicated, on the indexer in which file will the replicated lookup definitions appear?

0 Karma
1 Solution

Path Finder

I have figured out what's the problem. It turned out that:
- For the new lookup which I setup using the Web UI, it eventually (need to wait for a while) gets replicated to the search peers and is usable whenever the lookup is needed in the indexer.
- For the lookups which I migrated from an existing stand-alone Splunk, where the configurations are stored in $SPLUNK_HOME/etc/apps/search_migration_app (see here), they will also work on the search peer.
- In my case, it turns out the search_head-to-search_peer replication isn't working for one of the server, and thus I got the error.

View solution in original post

0 Karma

Path Finder

I have figured out what's the problem. It turned out that:
- For the new lookup which I setup using the Web UI, it eventually (need to wait for a while) gets replicated to the search peers and is usable whenever the lookup is needed in the indexer.
- For the lookups which I migrated from an existing stand-alone Splunk, where the configurations are stored in $SPLUNK_HOME/etc/apps/search_migration_app (see here), they will also work on the search peer.
- In my case, it turns out the search_head-to-search_peer replication isn't working for one of the server, and thus I got the error.

View solution in original post

0 Karma

SplunkTrust
SplunkTrust

yes, as part of knowledge bundle, the lookup definitions are available for indexers, but they can be blacklisted as part of your config. can you check if your have any config that overrides it? https://docs.splunk.com/Documentation/Splunk/7.2.3/DistSearch/Limittheknowledgebundlesize

0 Karma

Path Finder

These lookups are pushed to Indexers in form of knowladge bundles, So searches running on indexers will get the required detail.

0 Karma

Path Finder

Re: Also, only if the lookup is needed in the indexer [ for example to lookup/transforms the data as part of indexing pipeline], you need them in the indexer cluster.

But that's not what I experienced. The lookup works in my dummy search:

| stats count | eval name="joe" | lookup test-lookup name as name output id

However, if I change it to this, it will fail as the indexer tries to use the lookup (that I found out from the search.log on the indexer):

index=some_index | rex field=uri "[?&]name=(?<name>[^\s&]+)" | lookup test-lookup name as name output id

0 Karma

Path Finder

What is the error you are getting on UI? also, did you check if 'name' value is extracting correctly. It may be that the error is just coming because of issue in 'name' field value.

0 Karma

Path Finder

The error is:
[splunkindexer1] Streamed search execute failed because: Error in 'lookup' command: Could not construct lookup 'test-lookup, name, as, name, output, id'. See search.log for more details..

I can get it to work by using local=true
lookup local=true ...

0 Karma

Motivator

@patng_nw

what is the version of Splunk?

And also are you creating lookup in any app?

0 Karma

Path Finder

@vishaltaneja07011993 I am using 7.2.3, and I created lookup in Search using UI.

0 Karma

Motivator

@patng_nw

On all the search head cluster member is the lookup is replicated?

0 Karma

Path Finder

Yes, they're replicated on all the other search heads. I ran the same dummy search ( | stats count | eval name="joe" | lookup test-lookup name as name output id ) and they work on all search heads. It's only when the search involves the indexer then the lookup failed.

0 Karma