Splunk Search
Highlighted

Where does a lookup table need to be in a distributed search environment?

Contributor

All,

I'm having an issue where one of my indexers is complaining about a lookup table that I have setup on my search head. I get the error

[IndexerMachine] Streamed search execute failed because: Error in 'lookup' command The lookup table 'groupIdToName' does not exist.

From what I can tell from other Splunk Answers, the lookup table should be replicated to the search peers as part of the bundle replication (http://answers.splunk.com/answers/28541/lookup-table-does-not-exist.html). However, when I look at $SPLUNK_HOME/var/run/searchpeers/{most recent bundle} on the search peer, I don't see the lookup that should have been copied. In fact, I don't even see the system folder in that bundle.

So, I have two questions:

  1. How should a lookup be set up for a distributed search environment (i.e. should the lookup live on the search head, indexers, or both)?
  2. Assuming my set up is correct and the lookup should only live on the search head, how do I make sure that the lookup gets copied as part of the bundle replication?

Thanks!

0 Karma
Highlighted

Re: Where does a lookup table need to be in a distributed search environment?

SplunkTrust
SplunkTrust

Does the lookup have appropriate permissions in Search Head?

0 Karma
Highlighted

Re: Where does a lookup table need to be in a distributed search environment?

Contributor

Yes. It is a globally permissioned lookup stored in the $SPLUNK_HOME$/etc/system/lookup folder. Everyone has read access to the lookup.

0 Karma
Highlighted

Re: Where does a lookup table need to be in a distributed search environment?

SplunkTrust
SplunkTrust

Try moving it feom etc//system to etc//apps into any application.

0 Karma
Highlighted

Re: Where does a lookup table need to be in a distributed search environment?

Contributor

@somesoni2 - I moved the lookup to an application. I see it in the $SPLUNK_HOME$\var\run\searchpeers\$BUNDLE_FOLDER$\apps\myApp\lookups folder, but Splunk still shows the same error saying it wasn't found on the indexer.

Right now, my workaround is to use local=true for the lookup, but that's obviously not ideal. I'm not sure how to debug further.

0 Karma
Highlighted

Re: Where does a lookup table need to be in a distributed search environment?

SplunkTrust
SplunkTrust

I was referring to move it to $SPLUNK_HOME\etc\apps\lookups folder. This way it will be part of replication bundle. You can use search app for testing, if you don't want to create a new one.

0 Karma
Highlighted

Re: Where does a lookup table need to be in a distributed search environment?

Contributor

Right, I moved it there on the search head. My point is that it appears to be replicated (it shows up in the replication folder on the indexer). But I'm still getting the error.

0 Karma
Highlighted

Re: Where does a lookup table need to be in a distributed search environment?

Contributor

This was related to an app hitting an error when trying to perform bundle replication. The app was creating a file name that was way too long. Adding the app to the blacklist for bundle replication fixed the issue.

View solution in original post

Highlighted

Re: Where does a lookup table need to be in a distributed search environment?

Explorer

Where did you find what app was causing the problem?

0 Karma
Highlighted

Re: Where does a lookup table need to be in a distributed search environment?

Contributor

I don't remember, but I likely found it in the splunkd.log file. I'm sure there was an error there about bundle replication.

0 Karma
Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.