Where does a lookup table need to be in a distributed search environment?

[bruceclarke]

All,

I'm having an issue where one of my indexers is complaining about a lookup table that I have setup on my search head. I get the error

[IndexerMachine] Streamed search execute failed because: Error in 'lookup' command The lookup table 'groupIdToName' does not exist.

From what I can tell from other Splunk Answers, the lookup table should be replicated to the search peers as part of the bundle replication (http://answers.splunk.com/answers/28541/lookup-table-does-not-exist.html). However, when I look at $SPLUNK_HOME/var/run/searchpeers/{most recent bundle} on the search peer, I don't see the lookup that should have been copied. In fact, I don't even see the system folder in that bundle.

So, I have two questions:

1. How should a lookup be set up for a distributed search environment (i.e. should the lookup live on the search head, indexers, or both)?
2. Assuming my set up is correct and the lookup should only live on the search head, how do I make sure that the lookup gets copied as part of the bundle replication?

Thanks!

[bruceclarke · Accepted solution]

This was related to an app hitting an error when trying to perform bundle replication. The app was creating a file name that was way too long. Adding the app to the blacklist for bundle replication fixed the issue.

[nlembrechts]

Where did you find what app was causing the problem?

[bruceclarke]

I don't remember, but I likely found it in the splunkd.log file. I'm sure there was an error there about bundle replication.

[somesoni2]

Does the lookup have appropriate permissions in Search Head?

[bruceclarke]

Yes. It is a globally permissioned lookup stored in the $SPLUNK_HOME$/etc/system/lookup folder. Everyone has read access to the lookup.

[somesoni2]

Try moving it feom etc//system to etc//apps into any application.

[bruceclarke]

@somesoni2 - I moved the lookup to an application. I see it in the $SPLUNK_HOME$\var\run\searchpeers\$BUNDLE_FOLDER$\apps\myApp\lookups folder, but Splunk still shows the same error saying it wasn't found on the indexer.

Right now, my workaround is to use local=true for the lookup, but that's obviously not ideal. I'm not sure how to debug further.

[somesoni2]

I was referring to move it to $SPLUNK_HOME\etc\apps\lookups folder. This way it will be part of replication bundle. You can use search app for testing, if you don't want to create a new one.

[bruceclarke]

Right, I moved it there on the search head. My point is that it appears to be replicated (it shows up in the replication folder on the indexer). But I'm still getting the error.