Splunk Search

Where does a lookup table need to be in a distributed search environment?

bruceclarke
Contributor

All,

I'm having an issue where one of my indexers is complaining about a lookup table that I have setup on my search head. I get the error

[IndexerMachine] Streamed search execute failed because: Error in 'lookup' command The lookup table 'groupIdToName' does not exist.

From what I can tell from other Splunk Answers, the lookup table should be replicated to the search peers as part of the bundle replication (http://answers.splunk.com/answers/28541/lookup-table-does-not-exist.html). However, when I look at $SPLUNK_HOME/var/run/searchpeers/{most recent bundle} on the search peer, I don't see the lookup that should have been copied. In fact, I don't even see the system folder in that bundle.

So, I have two questions:

  1. How should a lookup be set up for a distributed search environment (i.e. should the lookup live on the search head, indexers, or both)?
  2. Assuming my set up is correct and the lookup should only live on the search head, how do I make sure that the lookup gets copied as part of the bundle replication?

Thanks!

0 Karma
1 Solution

bruceclarke
Contributor

This was related to an app hitting an error when trying to perform bundle replication. The app was creating a file name that was way too long. Adding the app to the blacklist for bundle replication fixed the issue.

View solution in original post

bruceclarke
Contributor

This was related to an app hitting an error when trying to perform bundle replication. The app was creating a file name that was way too long. Adding the app to the blacklist for bundle replication fixed the issue.

View solution in original post

nlembrechts
Explorer

Where did you find what app was causing the problem?

0 Karma

bruceclarke
Contributor

I don't remember, but I likely found it in the splunkd.log file. I'm sure there was an error there about bundle replication.

0 Karma

somesoni2
Revered Legend

Does the lookup have appropriate permissions in Search Head?

0 Karma

bruceclarke
Contributor

Yes. It is a globally permissioned lookup stored in the $SPLUNK_HOME$/etc/system/lookup folder. Everyone has read access to the lookup.

0 Karma

somesoni2
Revered Legend

Try moving it feom etc//system to etc//apps into any application.

0 Karma

bruceclarke
Contributor

@somesoni2 - I moved the lookup to an application. I see it in the $SPLUNK_HOME$\var\run\searchpeers\$BUNDLE_FOLDER$\apps\myApp\lookups folder, but Splunk still shows the same error saying it wasn't found on the indexer.

Right now, my workaround is to use local=true for the lookup, but that's obviously not ideal. I'm not sure how to debug further.

0 Karma

somesoni2
Revered Legend

I was referring to move it to $SPLUNK_HOME\etc\apps\lookups folder. This way it will be part of replication bundle. You can use search app for testing, if you don't want to create a new one.

0 Karma

bruceclarke
Contributor

Right, I moved it there on the search head. My point is that it appears to be replicated (it shows up in the replication folder on the indexer). But I'm still getting the error.

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.