topic Re: Where does a lookup table need to be in a distributed search environment? in Splunk Search

Where does a lookup table need to be in a distributed search environment?

bruceclarke — Tue, 02 Dec 2014 14:47:46 GMT

All,

I'm having an issue where one of my indexers is complaining about a lookup table that I have setup on my search head. I get the error

[IndexerMachine] Streamed search execute failed because: Error in 'lookup' command The lookup table 'groupIdToName' does not exist.

From what I can tell from other Splunk Answers, the lookup table should be replicated to the search peers as part of the bundle replication (http://answers.splunk.com/answers/28541/lookup-table-does-not-exist.html). However, when I look at $SPLUNK_HOME/var/run/searchpeers/{most recent bundle} on the search peer, I don't see the lookup that should have been copied. In fact, I don't even see the system folder in that bundle.

So, I have two questions:

How should a lookup be set up for a distributed search environment (i.e. should the lookup live on the search head, indexers, or both)?
Assuming my set up is correct and the lookup should only live on the search head, how do I make sure that the lookup gets copied as part of the bundle replication?

Thanks!

Re: Where does a lookup table need to be in a distributed search environment?

somesoni2 — Tue, 02 Dec 2014 17:31:56 GMT

Does the lookup have appropriate permissions in Search Head?

Re: Where does a lookup table need to be in a distributed search environment?

bruceclarke — Tue, 02 Dec 2014 17:44:30 GMT

Yes. It is a globally permissioned lookup stored in the $SPLUNK_HOME$/etc/system/lookup folder. Everyone has read access to the lookup.

Re: Where does a lookup table need to be in a distributed search environment?

somesoni2 — Tue, 02 Dec 2014 17:48:13 GMT

Try moving it feom etc//system to etc//apps into any application.

Re: Where does a lookup table need to be in a distributed search environment?

bruceclarke — Mon, 08 Dec 2014 22:10:49 GMT

@somesoni2 - I moved the lookup to an application. I see it in the $SPLUNK_HOME$\var\run\searchpeers\$BUNDLE_FOLDER$\apps\myApp\lookups folder, but Splunk still shows the same error saying it wasn't found on the indexer.

Right now, my workaround is to use local=true for the lookup, but that's obviously not ideal. I'm not sure how to debug further.

Re: Where does a lookup table need to be in a distributed search environment?

somesoni2 — Mon, 08 Dec 2014 23:26:26 GMT

I was referring to move it to $SPLUNK_HOME\etc\apps\lookups folder. This way it will be part of replication bundle. You can use search app for testing, if you don't want to create a new one.

Re: Where does a lookup table need to be in a distributed search environment?

bruceclarke — Mon, 08 Dec 2014 23:49:46 GMT

Right, I moved it there on the search head. My point is that it appears to be replicated (it shows up in the replication folder on the indexer). But I'm still getting the error.

Re: Where does a lookup table need to be in a distributed search environment?

bruceclarke — Mon, 22 Dec 2014 15:44:39 GMT

This was related to an app hitting an error when trying to perform bundle replication. The app was creating a file name that was way too long. Adding the app to the blacklist for bundle replication fixed the issue.

Re: Where does a lookup table need to be in a distributed search environment?

nlembrechts — Tue, 23 Jan 2018 16:35:41 GMT

Where did you find what app was causing the problem?

Re: Where does a lookup table need to be in a distributed search environment?

bruceclarke — Tue, 23 Jan 2018 16:39:32 GMT

I don't remember, but I likely found it in the splunkd.log file. I'm sure there was an error there about bundle replication.