Why do lookup searches fail with "The lookup table does not exist or is not available" error?

[dewoodruff]

Searches with lookups are failing in our environment. I have created a lookup file called dt1.csv and a lookup definition called dt1. Both the file and the definition have read and write permissions for all users in the search app.

This works successfully and shows the contents of the lookup:

|inputlookup dt1 

However, using the lookup as part of a query such as:

<search> |lookup dt1 cs_host

Fails with these errors:

[indexer1] Streamed search execute failed because: Error in 'lookup' command: The lookup table 'dt1' does not exist or is not available.
[indexer2] Streamed search execute failed because: Error in 'lookup' command: The lookup table 'dt1' does not exist or is not available.

Every indexer errors out with that message. This happens regardless of the app the lookup is created in, and regardless of the user that creates the lookup.

Any ideas on what would cause lookups to fail with these errors? We're on Splunk 6.5.1.

[dewoodruff · Accepted solution]

The root cause ended up being a custom app with very long file paths caused the knowledge bundle to fail to replicate. Once the app was removed, the knowledge bundle began replicating again and lookups started working.

The is apparently a known issue with 6.5.1 and has been resolved in some later version.

[aaraneta_splunk]

@dewoodruff - Glad you found the solution to your question. Please don't forget to click "Accept" to close out your question and upvote any answers/comments that were helpful. Thanks!

[evertonpsp]

Helo, Can You help me?

I have a like problem, but i think that the root cause is permission, since the problem happens for the users of a certain Role and the others work.

The file (Lookup table files) is read-only for all users and apps.

The Role Capabilities of the user that works is different from the Role Capabilities of the problem user, but I do not know which one I should add so that both work.

Does anyone have any ideas?
The problem can be Capabilities?

Thanks!

[DalJeanis]

Okay, so IIRC, "local" forces the lookup action to be executed on the search head, while the other does not. Doesn't that indicate that the lookup table is not being replicated to the indexers/peers?

https://answers.splunk.com/answers/343835/how-to-distribute-lookup-tables-in-an-indexer-clus.html
https://answers.splunk.com/answers/634/in-a-distributed-search-environment-where-do-my-configuration...

This thread describes a similar issue when the lookup table name was too long to be bundled. (obviously not the case here.)
https://answers.splunk.com/answers/200719/where-does-a-lookup-table-need-to-be-in-a-distribu.html

In any case, on several threads I saw the admonition, "Check your bundle replication error messages." Something in the replication process is screwy.

Lastly, the lesson learned in this one was that the new/altered lookup table had to be on the SH captain, not just any search head.
https://answers.splunk.com/answers/338008/why-do-i-see-old-data-in-my-lookup-table-in-a-sear.html

[adonio]

try and do your search and | lookup dt1.csv cs_host

[woodcock]

See the answer by @sjohnson. One way to test is to try this which will work but it will be slower than it should be:

 <search> |lookup local=true dt1 cs_host

[dewoodruff]

We're not able to find a distsearch.conf file in the search app directory. The query does succeed using local=true.

[woodcock]

Then you have a permission or (app) scope problem and you must not be running the 2 searches as the same user in the same app. Show me the URL for both |inputlookup dt1, <search> |lookup dt1 cs_host, and <search> |lookup local=true dt1 cs_host and make sure that you are logged in as the same user each time. It is surely that you are in 2 different apps; one which has access to the lookup and the other which does not.

[adauria_splunk]

They were run both as the same regular user, and as the same administrative user, with the same results. Everything was done within the search app only.

[dewoodruff]

Here you are. Hostname and index name removed for privacy.

inputlookup:

<hostname>/en-US/app/search/search?q=%7Cinputlookup%20dt1&display.page.search.mode=verbose&dispatch.sample_ratio=1&earliest=-24h%40h&latest=now&display.general.type=statistics&sid=1494609014.208099&display.page.search.tab=statistics

index= |lookup dt1 cs_host

<hostname>/en-US/app/search/search?q=search%20index%3D<indexname>%20%7Clookup%20dt1%20cs_host&display.page.search.mode=verbose&dispatch.sample_ratio=1&earliest=-24h%40h&latest=now&display.general.type=events&display.page.search.tab=events&sid=1494609057.208159

index= |lookup local=true dt1 cs_host

<hostname>/en-US/app/search/search?q=search%20index%3D<indexname>%20%7Clookup%20local%3Dtrue%20dt1%20cs_host&display.page.search.mode=verbose&dispatch.sample_ratio=1&earliest=-24h%40h&latest=now&display.general.type=events&display.page.search.tab=events&sid=1494609075.208172

[woodcock]

What version of splunk?

[woodcock]

Are you clustered?

[dewoodruff]

6.5.1. Search head is standalone. There are multiple indexers.

[sjohnson_splunk]

Is there a distsearch.conf in the search app that has a blacklist for lookups?

[adauria_splunk]

I am the SE assisting here. No, we can't seem to find any distsearch.conf outside of the default directories. I don't see anything in there blacklisting these lookups.