Deployment Architecture

Search Head Cluster: Lookups definitions not replicated to indexers

patng_nw
Communicator

I have a search head clusters with an indexer cluster, version 7.2.3.

On a search head, using Web UI I created a new file-based lookup in Search. On a search head I did a dummy search (which didn't involve the indexer) and made sure that the lookup is working. However, when I do a search which involves the indexer, the lookup failed.

On my indexer, I found that the lookup file was successfully replicated (I found it in a sub-folder under $SPLUNK_HOME/var/run/searchpeers/). I looked at the search.log in the indexer and saw that the indexer cannot find the lookup definition. So I think the lookup definition itself doesn't seem to be replicated.

Are lookup definitions replicated by default to the indexer? And if it's replicated, on the indexer in which file will the replicated lookup definitions appear?

0 Karma
1 Solution

patng_nw
Communicator

I have figured out what's the problem. It turned out that:
- For the new lookup which I setup using the Web UI, it eventually (need to wait for a while) gets replicated to the search peers and is usable whenever the lookup is needed in the indexer.
- For the lookups which I migrated from an existing stand-alone Splunk, where the configurations are stored in $SPLUNK_HOME/etc/apps/search_migration_app (see here), they will also work on the search peer.
- In my case, it turns out the search_head-to-search_peer replication isn't working for one of the server, and thus I got the error.

View solution in original post

0 Karma

patng_nw
Communicator

I have figured out what's the problem. It turned out that:
- For the new lookup which I setup using the Web UI, it eventually (need to wait for a while) gets replicated to the search peers and is usable whenever the lookup is needed in the indexer.
- For the lookups which I migrated from an existing stand-alone Splunk, where the configurations are stored in $SPLUNK_HOME/etc/apps/search_migration_app (see here), they will also work on the search peer.
- In my case, it turns out the search_head-to-search_peer replication isn't working for one of the server, and thus I got the error.

0 Karma

lakshman239
SplunkTrust
SplunkTrust

yes, as part of knowledge bundle, the lookup definitions are available for indexers, but they can be blacklisted as part of your config. can you check if your have any config that overrides it? https://docs.splunk.com/Documentation/Splunk/7.2.3/DistSearch/Limittheknowledgebundlesize

0 Karma

jvishwak
Path Finder

These lookups are pushed to Indexers in form of knowladge bundles, So searches running on indexers will get the required detail.

0 Karma

patng_nw
Communicator

Re: Also, only if the lookup is needed in the indexer [ for example to lookup/transforms the data as part of indexing pipeline], you need them in the indexer cluster.

But that's not what I experienced. The lookup works in my dummy search:

| stats count | eval name="joe" | lookup test-lookup name as name output id

However, if I change it to this, it will fail as the indexer tries to use the lookup (that I found out from the search.log on the indexer):

index=some_index | rex field=uri "[?&]name=(?<name>[^\s&]+)" | lookup test-lookup name as name output id

0 Karma

jvishwak
Path Finder

What is the error you are getting on UI? also, did you check if 'name' value is extracting correctly. It may be that the error is just coming because of issue in 'name' field value.

0 Karma

patng_nw
Communicator

The error is:
[splunkindexer1] Streamed search execute failed because: Error in 'lookup' command: Could not construct lookup 'test-lookup, name, as, name, output, id'. See search.log for more details..

I can get it to work by using local=true
lookup local=true ...

0 Karma

vishaltaneja070
Motivator

@patng_nw

what is the version of Splunk?

And also are you creating lookup in any app?

0 Karma

patng_nw
Communicator

@vishaltaneja07011993 I am using 7.2.3, and I created lookup in Search using UI.

0 Karma

vishaltaneja070
Motivator

@patng_nw

On all the search head cluster member is the lookup is replicated?

0 Karma

patng_nw
Communicator

Yes, they're replicated on all the other search heads. I ran the same dummy search ( | stats count | eval name="joe" | lookup test-lookup name as name output id ) and they work on all search heads. It's only when the search involves the indexer then the lookup failed.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...